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FOREWORD 



When the U.S. Office of Eaucation was chartered in 1867, 
one charge to its commissioners was to determine the 
nation's progess in education. The National Assessment of 
Educational Progress (NAEP) was initiated a century later to 
address, in a systematic way, that charge. 

Each year since 1969, ffetional Assessment has gathered 
information about levels of education achievement across the 
country and reported its findijigs to the nation. NAEP 
surveys the education attainments of 9-year-olds, 
13-year-olds and 17-.year-olds in 10 learning areas: art, 
career and occupational development, citizenship, litera- 
'ture, mathematics, music, reading, ccience, social studies 
and writing. Different learning areas are assessed every 
year, and all areas are periodically reassessed in order to 
measure possible changes in education achievement. National 
Assessment has interviewed and tested more than one million 
young Anericans since 1969. 

Learning-area assessments evolve from a consensus 
process. Each assessment is the product of several years of 
work by a great many educators, scholars and lay persons 
from all over the nation. Intially, these people design 
objectives for each subject area, proposing general goals 
they feel Anericans should be achieving in the course of 
their education. After careful reviews, these objectives 
are given to exercise (item) writers, whose task it is to 
create measurement instruments appropriate to the objec- 
tives. 

When the exercises have passed extensive reviews by 
subject-matter specialists, measurement experts and lay 
persons, they are administered to probability samples. The 
people who conpose these samples are choosen in such a way 
that the results of their assessment can be generalized to 
an entire national popualtion. That is, on the basis of the 
performance of about 2,000 9-year-olds on a given exercise, 
we can make generalizations about the probable performance 
of all 9-year-olds in the nation. 
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After assessment data have been collected* scored and 
analyzed. National As^es^ent publishes reports to dissemi- 
nate the results as widely as possible. Not all exercises 
are released for publication. Because NAEP will readmin- 
ister some of the same exercises in the future to determine 
lAiether the performance level of Americans has increased, 
remained the same or decreased, it is essential that they 
not be released in order to preserve the integrity of the 
study.' 
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THE 1981-82 ASSESSMENT OF MATHEMATICS AND 
CI,TIZENSHIP/SOCIAL STUDIES 



Overvl-ew 

The National Assessrnent of Educational Progress (NAEP) 
completed its ^third assessment of mathematics and citizen- 
ship/social studies during the 1981-82 school year. Mathe- 
matics was assessed previously in 1972-73 and 1977-78; citi- 
zenship was first assessed in 1969-70 and social studies in 
1971-72; both were assessed for a Second time in 1975-76. 
Because there is a high degree of similarity between the 
goals considered important in citizenship and social studies 
education, NAEP combined the two into one comprehensive area 
— citizenship/ social studies — for the 1981-82 assessment. 

Each assessment surveyed the achievement of Anerican 9-t 
13- and 17-year-old students using a deeply stratified, 
multistage probability sample. To measure changes in 
performance between assessments, some of the exercises 
administered in the previous assessments were readministered 
under virtually identical administrative conditions. Since 
1969, • National Assessment has conducted major assessments in 
art, career and occupational development, citizenship, 
literature, mathematics, music, reading, science, social 
studies, and writing and has surveyed several other learning 
areas on a smaller scale. All of the major areas except 
career and occupational development have been reassessed one 
or more times. In some of the earlier assessments, 
17-year-olds who were no longer in school and young adults 
ages 26-35 were also surveyed. Learning areas and ages 
assessed since 1969 are shown in Exhibit 1. 

Although science. was also assessed in 1981-82, this 
procedural handbook discusses only the mathematics and citi- 
zenship/social studies assessments. Though data were 
collected by National Assessment, the science assessment was 
sponsored by the National Science Foundation, and conducted 
through a contract with the U&iiversity of Minnesota. 
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EXHIBIT 1 

Learning Areas and Ages' Assessed From 1 969 to 1982 



Assessment Year/Learning Areas 



1 1969-70 



Science 
Writing 
Citizenship 



2 1970-71 



Reading 
Literature 



Ages Assessed* 
9 13 I7IS 170S Adult 



X 
X 
X 



X 
X 



X 
X 
X 



X 
X 
X 



X X 
X X 



X 
X 
X 



X 
X 



X 
X 
X 



X 
X 



3 1971-72 



Music 

Social Studies 



^ 1972-73 



ScienGe(2) 
Mathematics 



5 1973-74 



Career and Occupational 

Developnent 
Writing(2) 



6 197'<-75 



Reading(2) 
Art 



7 1975-76 



Citizenship(2)/, 

Social Studies(2) 
Mathematics** 



8 1976-77 



Science( 3) 

Basic Life Skills** 

Health** 

Ehergy** 

Peading**(2) 

Science**(3) 



X X 
X X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X X 
X X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 
X 
X 



lu 
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EXHIBIT 1 
Learning.. (Continued) 



\ 



Assessment Year/Learning Areas 




9 1971-78 



thematics(2) 
Cinsuner Skills** 




t(2) 
Music(2) 
WritingO) 



11 1979-80 



Mathematics(3) 
Citizenship/ 

Social Studies(3) 
Science»»(4) 



NOTES: 



Ages- Assessed* 
9 13 17IS 170S Adult 



X 
X 

X 



X 
X 
X 



X 
X 



X 
X 
X 





Reading (3)/ 


X 


X X 




Literature (2) 


X 


X X 




Art (2) 




X 


12 


1980-81 








No data collection 






13 


1981-82 







X 
X 



X 
X 



X 
X 



X 
X 



«» 



( ) 



17IS denotes 17-year-olds enrolled in public or 
private schools; I7OS dfenotes 17-year-olds who 
dropped out of school or graduated prior to the time 
of the assessment. 

Indicates small, special-interest probe assessments 
conducted on limited samples at specific ages. 
Shows second and subsequent assessments of an area. 



li 



PAGE 8 



Objectives 

The primary goal of the National Assessment of Educa- 
tional Progress is to report on the current educational 
status of young Anericans and to monitor any changes in 
achievement over time. For each learning area to be 
assessed , NAEP asks consultants to review, revise and/or 
develop objectives that define the subject area. Since the 
objectives provide guidelines for exercise writers, consult- 
ants are asked to include examples of the knowledge, skills, 
understandings and attitudes to be assessed at each age 
level . 

Education in Anerica is a collaborative enterprise 
involving a great many people with widely differing philoso- 
phies . Provid ing information about education nationwide 
would be considerably . easier if there were complete agree- 
ment about the means and ends of Anerican education. The 
fact is, however, that' Anericans have conflicting and some- 
times contradictory values regarding the goals of education 
and the means for achieving them. To have an assessment 
that is truly national in scope and takes into account the 
diversity of curricula, values and goals across the country, 
National Assessment employs a consensus process for devel- 
oping objectives. 

Several types of consultants help to develop Kational 
Assessment objectives. College and university specialists 
in a learning area insure that the objectives include impor- 
tant concepts that the schools should be teaching. Educa- 
tors, including classroom teachers, curriculun supervi;.ors 
and persons involved in teacher education, make sure 'hat 
the objectives describe concepts, skills and attitudes that 
the schools should be teaching and those that they presently 
are teaching. Concerned citizens, jparents and other Inter- 
ested lay persons must agree that the objectives are impor- 
tant for young people to achieve, are free from educational 
Jargon and are not biased against or offensive to any 
groups. Consultants are selected to represent different 
regions of the country, community types and minority groups. 
Ihey also represent a range of experience with students of 
different ages. Thus, the exercises used t ' report informa- 
tion about mathematics and citi zenshl p/ social studies 
achievement measure broad education objectives that >^epre- 
sent a consensus of educators', subject-matter experts' and 
interested lay persons' opinions about what young Anericans 
should know and be able to do. These objectives are not an 
attempt to mandate behavior and value systems; rather, they 
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3 

represent goals that a diverse group of people have 
identified as desirable for* young Americans to accomplish. " 



Mathematics CbJ ectiyes 

To measure the status of mathematics in 1981-82, objec- 
tives used in the 1977-78 assessment were reviewed by 
ntanerous mathematicians and mathematics educators and 
revised-.,^ — .-Ihe basic framework of the earlier objectives was 
retained for the 1981-8*2 assessment. Revisions reflect 
current content and trends in school mathematics.. 

The mathematics objectives are structured as a two-dimen- 
sional content-by-process matrix. For the 1981-82 assess- 
ment, the process dimension of the matrix includes five 
cognitive levels — knowledge, skills, understanding, appli- 
cation and problem solving, and attitudes. Ihe content 
dimension contains six categories — nunbers and nuneration ; 
variables and relationships; geometry (size, shape and posi- 
tion); measurement; statistics arxJ probability; and tech- 
nology. 

Changes in the objectives from the second to third 
assessment included the following: 

1. Expansion of the content by-cognitive-process matrix 
from four to five process levels and from five to six 

.content levels. Attitudes were included as the fifth 
process level and "technology" (including computer 
literacy and calculators) was added as the sixth 
content area. * 

2. Addition of estimation objectives, including mental 
computation, computational estimation and problem 
solving estimation • 

3. Increased emphasis on probability and statistics and 
problem solving. ' 

New. exercises were developed to provide coverage of the 
revised objectives, and exercises kept secure from previous 
assessments were used to measure changes in achievement 
across tjme* For further details about the objectives used 
to assess math^natics in 1981-82, see Mathematics Cbjeo- 
tives, 1981-82 Assessment , (1981). \ 



13 



PAGE 10 



Citizen ship/ Social Studies Cbjectlves 

In preparation for development of objectives for the 
1981-82 citizenship/ social studies assessment, an initial 
docunent combining previously developed NAEP objectives in 
citizenship and social studies as well as other objectives 
developed in each of these areas was prepared in September, 
1978. It was subjected to numerous revisions based on the 
reviews of several hundred people representing diverse 
regional, ethnic, cultural and educational interests. 

The final objectives express a variety of perspectives 
regarding the goals of citizenship/ social studies education. 
The content does not promote any particular educational, 
political or social viewpoint, but draws freely on all the 
major social science disciplines. The objectives incorpo- 
rate o variety of learning approaches, but they do not 
/ prescribe any one method . Ihroughout , the objectives 
display a concern for political and cultural differences by 
stressing the importance of an interest in hunan interrela- 
tionships and a concern for basic human rights. 

The National Assessment citizenship/ social studies objec- 
tives are stated as general goals. Not every objective is 
appropriate for each of the three age levels. Good citizen- 
ship does not require 100? mastery of all of these goals. 
Yet achievement of the knowledge, skills, attitudes and 
behaviors that make up these goals should result in well- 
informed citizens and decision makers. 

New exercises were developed to measure new citizenship/ 
social studies objectives, and items from previous assess- 
ments were used to measure changes in achievement across 
time. For more information about the citi zen ship/ social 
studies objectives, ' see Citizenship and Social Studies 
Objectives, 1981-82 Assessment , ( 1980). 
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Development of Exercises 

Following development of objectives, exercises (items) 
were created to measure achievement of those objectives. 
Each exercise was designed so. that its results could either 
be used alone, as an indicator of performance on a specific 
task, or used in conjunction with results from other ex er- 
ci5es to give a more general picture of achievement levels. 

Assessments cannot be developed directly from objectives. 
First, the overall structure of the item pool must be 
outlined. Items from previous assessments are examined to 
determine which will be used to detect changes in perform- 
ance and whether they will be categorized according to the 
original objectives scheme or reorganized in a different 
classification system. To measure new objectives, decisions 
must be made as to whether all objectives will bo measured, 
how many exercises will be used to measure each objective 
and the specific contents and formats appropriate to each 
objective. 

Following the definition of the assessment's structure, 
items are developed to flesh out that structure. Items are 
developed,- reviewed and revised in several iterations to 
ensure that they measure the intended objective, are meas- 
ures of important concepts and are free from bias. 



Mathematics Exercise Development 

Items for the 1981-82, mathematics assessment were created 
through a network of development centers. Numerous consult- 
ants, including university mathematics educators, classroom 
teachers and lay citizens, were involved in the development, 
review and revisions of the items. 

The development centers located at were colleges, univer- 
sities or curriculum/ research centers that employed a nunber 
of prominent mathematics educators willing to work together 
over several months. to develop a portion of the exercises 
for the assessment. Such centers proved successful in the 
development of exercises for the second mathematics assess- 
ment and thus were used again for the third survey. 

The centers provided numerous advantages both to NAEP and 
the writers. Using centers rather than conferences meant 
lower costs because travel was reduced. Consultants were 
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able to integrate work on NAEP exercise developnent vdth. 
their other professional activities, causing a minimun of 
disruption and increasing their efficiency. 

In the development and selection of mathematics exercises 
for the 1981-82 assessment, care was taken to ensure an 
appropriate balance of emphasis on both the content and 
process dimensions of the objectives matrix. This balance 
was achieved through attention to a questions organized 
according to the categories of the process dimension, ques- 
tions based on the combined priorities of the interested 
public , mathematicians, mathematics educators and educa- 
tional administrators. These questions appear on pages 
16-18 of Mathematics Objectives, 1981-82 Assessment. 

The content domain for the 1981-82 assessment of mathe- 
matics drew primarily from the current curriculun of elemen- 
tary and secondary schools, although some projections of 
future mathematics emphases were acknowledged (for example, 
assessment of problem solving strategies and use' of calcula- 
tors and computers). Mathematics up to but not including 
calculus was included in the assessment exercises; however, 
a 3tudent v*io had no more mathematics than a first-year 
algebra course should have been able to answer most of the 
exercises . 

Exercises were classified according to the six content 
categories, as shown in Exhibit 2. These content categories 
were not intended to be represented equally in the assess- 
ment. The planned weighting of exercises by content is 
indicated in Exhibit 2. 



The five process domain categories for the third assess- 
ment are shown in Exhibit 3. Although each category 
suggests a type of mental process, neither objectives nor 
exercises fall neatly into a single process category because 
different students may use different processes or different 
combinations of processes. Thus, any system of process 
categories necessarily involves some arbitrary distinctions. 
Such a system is helpful, however, in considering the diver- 
sity of process levels possible within a given content 
category. The percentages in Exhibit 3 give the planned 
weighting of exercises by cognitive process levels. 
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" EXHIBIT 2 

Target Percentages of Exercises by Age and Content* to be 
Used In uhe 1981-82 Assessment of Mathematics 



Age 9 Age 13 Age 17 

A. Numbers and ^0% l\0% 35% 

numeration 

B. Variables and 10 10 20 

relationships 

C. Siape, size and 10 10 10 

position 

D. Measurement 15 15 10 - 

E. Probability and 5 5 10 

statistics 

F . Tec hn ol og y 10 10 10 

*These percenta^3^d^""*not add to 100% because the 
attitudinal exi^cises are not i.tcluaed. 



Development center consultants first conducted a logical 
analysis of the mathematics content area for which they were 
developing items. Centers used the revised objectives to 
inake necessary changes in content or emphasis in tfheir area. 
Using the secure exercises as the core of the set, new exer- 
cises were developed to either replace released exercises or 
to cover new topics. Centers received instructions as to 
the nunber of exercises ♦ age overlaps and process -^level 
based on the suggested weighting of the qbj^^ ix . 

After the first pass at exercise development was 
completed* the exercise set*wa^ sejrit out to external 
reviewers for comments. Their comments were reviewed by 
center s.taff » editorial changes were ma^ and the final sets 
with supporting documentation were sent to NAEP for field 
testing. , ' , 
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EXHIBIT 3 

Target Psrcentages of Ebcercises by Age and Process Level to 
be Used in the 1981-82 Assessment of Mathematics 



Age 9 .Age 13 Age 17 

I. Mathematical 20% ^5% ^5% 

knowledge 

II. Mathematical 25 25 25 

skill 

III. Mathematical 20 25 25 

understanding 

IV. Mathematical 25 30 30 

application 

V. AttitvxJes toward 10 5 5 

mathematics 



In addition to reviews at the development centers and 
mail reviews conducted by the centers, conferences were held 
at NAEP to review all of the exercise sets. Eight to ten 
consultants, representing teachers, university specialists 
and the development centers, were invited to Denver to 
review and edit the exercises. In addition to these 
reviews, the exercises were mailed out to an additional 
group of consultants. All comments were synthesized by NAEP 
staff and changes were made where appropriate. 



Citizenship/Social Studies Exercise Development 



Early in the planning stages for the develdpiferfE~o"f~citi^ 
zenship/ social studies, it was decided to survey broadly the 
entire dcxnain of citizenship and social studies as defined 
by the ' -jectives. No specific parameters were placed on 
the nUj. *<;r of exercises needed to measure each objective, 
but an attempt was made to cover all objectives. It was 
decided that, to the extent i^ssible, all secure exercises 
from previous assessments of citizenship and social studies 
would be used to measure changes in achievement over time. 
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Several factors influenced the approach used to develop 
exercises for the 1981-82 assessment 'Of citi zenshi p/ social 
studies. Previous developments in this subject area had 
been carried out through subcontracts or individual 
contracts- vdth consultants. Establishing development 
criteria and stringently monitoring the-application of the 
criteria had proven difficult at best during previous devel- 
opments, • Given the nunber of new objectives requiring 
development, the controversial nature of the subject area 
and the general lack of specificity of the objectives, the 
problems of long-distance monitoring by KA?!P staff seemed 
even greater . 

Successful exercise development in a short time called 
for a development procedure that could start up almost imme- 
diately, adapt as events unfolded and be closely monitored 
by the NAEP staff. Since staff members had been involved in 
all phases of the objective development and review phases, 
they were in the best position to assist the exercise 
writers and to help them translate the objectives into mean- 
ingful exercises. Based on these factors, it was decided to 
develop exercises using primarily local consultants who 
worked directly with the NAEP staff, 

A draft of the objectives that could be used for exercise 
development was first available in January, 1979. This 
draft was used to develop exercises during the first of 
three development phases. About 900 to 1300 new exercises 
were needed to adequately cover the objectives. More than 
800 exercises were developed by exercise writers during the 
first phase," from January through March 1979- IXiring the 
period from August to September 1979f approximately 200 
additional exercises were developed. After field testing 
and consultant reviews of exercises, a second development 
phase was initiated in August, 1979. The phase was intended 
to fill gaps in the exercise coverage of the objectives. 
Based on the losses during reviews and consultant sugges- 
tions, exercises were needed that reflected a greater diver- 
sity of experience in the United States from a multi-cul- 
tural, multi-ethnic ancj iglobal perspective. 

Based on decisions made by the Citizenship/ Social Studies 
Advisory Committee and the results of review conferences 
held in December 1979» a third development phase occurred in 
the first three months of 1980, to fill more gaps in 
coverage. 
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Field Testing of Exercises 



After exercises for mathematics and cltizenshl p/ social . 
studies had been developed, critiqued and revised, they were 
reviewed by NAEP staff. The results of these reviews were 
compiled and once again the exercises were revised. Exer- 
cises that survived the reviews were field-tested In schools 
across the country to discover problems In wording, direc- 
tions or administrative procedures and to collect exercise 
statistics, timing Information and scoring Information. 
Schools were selected to represent high- and low-Income 
communities as well as more typical communities. The field 
tests were administered to students In at least four class- 
rooms (approximately 100 students) at each of the ages 
assessed. 

So that field tests closely simulated actual assessment 
field procedures, the students recorded their answers in the 
exercise booklets, directions and questions were read to 
students from an audio tape and National Assesanent staff 
members, rather than classroom teachers, administered the 
tests. The students' responses to the exercises, as well as 
the administrators' reports of any field problems, helped 
both staff- and consultants to evaluate and revise the exer- 
cises. Revised exercises were sometimes field tested again. 

After exercises were field-tested, the results were 
reviewed by National Assessment staff and panels of subject 
area experts, educators and lay persons from across the 
country. Exercises were reviewed for age appropriateness by 
teachers who taught students at that age. Lay citizens, 
representing a variety of occupations and Interests, also 
reviewed the exercises to check for sex or racial/ethnic 
bias arid to consider the general Importance of each exer- 



A panel of subject area specialists then worked with NAEP. 
staff to make final selection of exercises to be used in the 
1981-82 assessments of mathematics and citi zenshi p/ social 
studies from the pool of exercises judged acceptable by the 
review panels. 



else . 
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Preparation of Asses^TOent Materials 

A total of 35 booklets were used to assess mathematics 
and citizenship/ social studies in 1981-82 seven for 
g-year-olds and fourteen for each 13- and 17-year-olds. Six 
of the seven booklets for 9-year-olds contained., exercises 
from both subject areas, and one booklet contained mathe- 
matics only. At ages 13 and 17f separate booklets were used 
for each subject area. For the two older ages, six booklets 
contained citizenship/ social studies exercises, and eight 
contained mathematics Booklets were constructed separately 
for each age level, since students at different ages 
received different sets of exercises . Thus, exercises for 
9-year-olds were not sequenced in the same order as those 
for 13-year-olds, and so forth. 

The following constraints were observed in preparing 
exercise booklets: 

- Each booklet contained exercises of varying difficulty 
so that respondents would not become bored by many easy 
exercises or discouraged by many difficult ones. 

- Exercises could nbt cue other exercises. In other 
words, the answer to one exercise could not he 
contained in another exercise in the same booklet. 

- Each booklet was timed so that It would take no more of 
a respondents* time than 45 minutes, the length of a 
typical class period. Booklets contained approximately 
30-35 minutes of exercise time and an additional 10-15 
minutes of Introductory material , instructions and 
b ac kg ro und que s 1 1 on s . ^ 

- Booklets were designed .to be similar. Insofar as 
possible, with respect to the number of different 
objectives and subobjectlves measured . Ebcerclses meas-r 
urlng a particular objective were scattered throughout 
the booklets so that many different respondents would 
respond to exercises related to a particular obj^ectlve. 

National Assessment makes every effort to minimize diffi- 
culties connected with the testing situation do that results 
will be, as nearly as possible, an accurate reflectiorf of 
what respondents know and can do. For example, space was 
provided for respondents to write their answers directly in 
the assessment booklets. Instead of on separate answer 
sbejiJtSi^_ It was felt, that this procedure would reduce 
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possibilities for error, especially for younger respondents. 
Considerable effort was also devoted to developing clear 
instructions for procedures the respondents should use in 
responding to an exercise. To minimize guessing, respon- 
dents were encouraged to select the "I don't know" response 
option included with most multiple-choice exercises or to 
write "I don't know" on the answer line for open-ended exer- 
cises if they felt they did not know the answer. 



Sampling 

Overview of the Sample Design 

National Assessment uses a deeply stratified, three-stage 
national probability sample design with oversampling of 
low-income and rural areas. The sample is designed and 
selected by the staff of the Research TV-iangle Institute 
(RTI), Raleigh, North Carolina, and monitored by NAEP staff. 

In the first stage of sampling, the IJhited States is 
divided into geographical units of counties or groups of 
contiguous- counties meeting a minimun population size 
requirement. These units, called primary sampling units 
(PSUs) , are stratified by both region and size of community. 
From the list of PSUs, a sample of PSUs that represents all 
regions and sizes of communities is drawn (without replace- 
ment) with probability proportional to population size meas- 
ures. Oversampling of low-income and rural areas is first 
performed at this stage by adjusting the estimated popula- 
tion size measures of those areas to increase sampling 
rates. Census Bnployment Survey data are used within large 
urban PSUs to further delineate and oversample low-income 
areas. Counties with high proportions of rural families are 
also over sampled. 

In the second stage, all public and private schools 
within each PSU selected in the first stage are listed. 
Schools within each PSU are selected (without replacement) 
with probabilities proportional to the estimated nunber of 
age-eligibles in the school. 

Ihe third stage of sampling occurs during the data 
collection period.^ A list pf all age-eligible students 
within each selected school is made. A simple random selec- 
tion of eligible students (without replacement) is obtained, 
and exercise, booklets are administered to the selected 
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students. Specially trained assessment employees select the 
sample and administer the booklets. 

Each respondent in the sample does not have the same 
probability of selection because some sub populations are 
over-sampled and because adjustments are made to compensate 
for student nonresponde and for some schools' refusal to 
participate. The selection probability for each individual 
is computed, and its reciprocal is used to weight each 
response in statistical calculations computed by NAEP to 
compensate for unequal rates of sampling and to insure 
proper representation in the population structure. 

The nunber of PSUs , schools within PSUs and students 
within schools is determined by optimun sampling principles. 
That is, a sample design is .a^ that will achieve the 

maximun precision for a given level of resources. 

At ages 9 and 13, almost all of the noninsti tutionalized 
population is enrolled in public and private schools. By 
age 17, however, up to 15% of the population has either 
dropped out or graduated ft^om secondary school. When 
possible, out-of-school 17-year-olds are assessed in order 
to provide comparable coverage of the 9-f 13- and 
17-year-old populations. Assessment of out-of-school 
17-year-olds and young adults ages 26-35 is quite expensive, 
however, and in 1981-82, funds to assess these groups were 
not available. 



Ihe 1981-82 Sample Design 

In March 1979, primary samples were selected to serve 
several assessments, beginning with the 1979-80 assessment. 
In the first stage of sampling, counties and county-equiva- 
lent independent cities or clusters of cities were strati- 
fied by region and by size of community. A total of 1,069 
primary sampling units were included in the sampling frame. 

In the first stage of sampling, the PSUs were stratified 
by. the four geographic regions defined by the Office of 
Business Economics, U.S. Department of Commerce (see REGION 
in Appendix 1). These regions are labelled Northeast, 
Southeast, Central and West. 

Within each region, PSUs were classified into five size- 
of-commuriity (SOC) categories. The sampling size of commu- 
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nity (SOC) variable has a year-specific definition. The 
size-of-community categories for the I98I-82 assessment were 
defined as follows : 



SOC 1: PSUs corresponding to U.S. Bureau of Census Standard 
Metropolitan Statistical Area (SMSA) counties 
containing all or part of a central city of 200,000 
or more population ("big city") in 1970. 



SOC 2: Remaining counties in "big city" SMSAs. 

SOC 3: Other counties containing all or part of a place 
with 25,000 or more population in I97O. 

SOC ^: Counties not qualifying for SOC 1, 2 or 3 and not 
classified as "extreme rural" (SOC 5). 

SOC 5: Counties not classified as SQC 1, 2 or 3, not having 
10,000 or more total I97O urban population, having 
some farm employment, and having relatively high 
values on an "extreme rural" index that was computed 
based on county labor force occupational classifica- 
tions. 

Thus, the design defines 20 primary strata (four geographic 
regions crossed with five SOC categories). 

Next, a size measure was ^associated with each PSU within 
each stratum. This size measure was based on student 
enrollment information. PSUs that included census-defined 
poverty areas and census-defined extreme rural areas were 
identified, and the size associated with these specially 
defined areas was doubled to ensure adequate representation 
of these groups . 

Before the sample was drawn, the PSUs within each stratum 
were ordered in a serpentine fashion by state. Increasing 
and then decreasing values of percent racial minorities were 
alternated by state. 



From the ordered sampling ft-ame, five equal size samples 
were selected (using a probability minimum replacement 
algorithm) with probability proportional to size. Each 
sample included 64 first-stage sampling units. Four of the 
samples were randomly assigned to various assessments. , Ihe 
fifth sample was reserved' as a source of replacements for 
primary units that refused to participate and a possible 
supplementary sample for a special study. 
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Within the primary strata . public and private schools 
were listed and further stratified by the estimated number 
of students in a school who were eligible at each age. 
anall schools were clustered until they formed a large 
enough group to respond to the same nunber of exercise book- 
lets as the larger schools in a stratu»i. Schools or school 
clusters were selected (without replacement) with proba- 
bility proportional to the nunber of age-eligibles in the 
school or cluster of schools. Once schools were identified, 
districts were contacted to check for changes in grade range 
and for the existence of new schools. This information was 
used to revise probabilities of schools* selection. The 
nunber of PS Us and the total nunber of schools in which 
assessment sessions were conducted in 1981-82 by age are 
shown in Exhibit 4. Assessment sessions in 1981-82 included 
mathematics and citizenship/ social studies booklets as well 
as science booklets. 



EXHIBIT ^ ' 

Number of Primary Sampling Units and Total Number 
of Schools Participating in 1981-82 by Age 



Age No* of PSUs No. of Schools 

9 64 413 

13 64 540 

17 64 396 



Students were then selected (without replacement) with 
equal probability in each sampled school. The target popu- 
lations for the assessment included 9-. 13- and 17-year-olds 
enrolled in either public or private schools at the time of 
the assessment who were not functionally handicapped to the 
extent that they could not participate in an assessment. 
Definitions of the age groups are: 9-year-olds were born 
during calendar year 1972; 13-year-olds were born during 
calendar year 1968; and 17-year-olds were born from October 
1, 1964 through September 30, 1965. ^ecific groups 
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excluded were non-English speaking, persons physically or 
mentally unable to respond, and persons in institutions or 
attending schools established for the physically or mentally 
handicapped. The nunber of students selected was propor- 
tional to the nunber of age eligibles with oversampling in 
low-income and rural areas. 



Data Collection 

A professional data collectior staff from the Research 
TViangle Institute (RTI), Raleiih, North Carolina adminis- 
tered the assessment booklets. This staff was used instead 
of school personnel to minimize ".he burden on participating 
schools and to ensure, as far )ossiblc, uniform adminis- 
tration conditions across the ^ry. NAEP staff worked 
closely with the subcontractv. ^ ensure adherence to 
rigorous administrative standards. 

Participation in the National Assessment is voluntary; 
however. National Assessment makes every effort to encourage 
schools selected in the sample to participate in the assess- 
ment. If ^ less than 75X of the selected 17-year-olds were 
available for a regular assessment session, a follow-up 
session was held in an effort to have a higher rate of 
completion by 17-year-old students. High rates of school 
and student cooperation were obtained in 198l-82f as shown 
in Bchibit 5. 



EXHIBIT 5 

School Cooperation Rates and Average Student Completion 
Rates in 1981-82 by Age 



Average Student 
Percent of Eligible Completion Rates in 

Age Schools Participating Participating Schools 

90.5? 
85.5? 
7^.2? 



9 88.3? 
13 , 89.2? 
17 86.5? 
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Approximately 14,000 9-year-.olds , 23,000 13-year-olds and 
23,000 IT-year-olds participated in the 1981-82 assessment 
of mathematics and citi zen ship/ social studies. Because 
National Assessment reports results for groups of students 
instead of individuals, it is not necessary for each -student 
to answer every exercise (item). Each student completed 
only one exercise booklet of about ^5 minutes in length. 
Approximately 2,000 students at each age responded to each 
mathematics booklet; approximately 2,000 9-year-olds and 
1,200 13- and 17-year-olds responded to each citi zen ship/ so- 
cial studies booklet. The nunber of booklets, per age and 
the average nunber of respondents assessed per booklet is 
indicated in Exhibit 6. 



EXHIBIT 6 

Number of ffethematics and Citizenship/ Social Studies 
Booklets and Average WJmber of Respondents Per Booklet 
in 1981-82 by Age 



Citizenship/ . 

Social Studies 
Mathematics 

Citizenship/ 

Social Studies 
Mathematics 



Age 



13 
13 



Number of 
Booklets 



6» 

7» 



Average Number 
of Respondents 
Per BDOklet 



,1990 
/2006 



121 1 
1970 



Citizenship/ 

Social Studies 17 
Mathematics 17 



6 

f 



1125 
2040 



Seven booklets were administered/ to 9-year-olds. In 6 
other booklets, -approximately 1/3 of the exercises 
surveyed citizenship/social studies and 2/3 were mathe- 
matics items. One booklet contained only mathematics , 
exercises. 
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In 1981-82, all booklets were administered to groups of 
10-25 students • Ihe groups varied in size depending on the 
nunber of eligible students and on an estimate of the 
nonresponse rate for a particular school. Allowing variable 
group sizes within schools enabled Nationr"* Assessment to 
obtain desired sampl^^ sizes in schools hawiug characteristi- 
cally low response ^'rates. Ihis feature also permitted 
last-minute modifications and adjustments to selection prob- 
- abilities necessitatefj by enrollment changes. 

Respondents within each age group were assessed at 
approximately the same time in the school year: 
13-year-blds from October to December, 9-year-olds in 
January and February and 17-year-olds in March and April, 

National Assessment takes steps to guarantee the 
anonymity of each respondent. Students' names were listed 
with 'their booklet identification nunber so that scoring and 
processing personnel could go back to the school lists for 
data verification — for instance, of background information 
— if necessary. These lists did not le^ve the schools and 
were destroyed six months following the assessment in a 
school. 

Paced audio tapes were used to insure uniform assessment 
conditions across the countrV and to move respondents 
through the booklets at the same speed. All directions for 
answering exercises were read on the tape. For most of the 
exercises the written portions of the exercise and the 
response options were read aloud to reduce the effect of 
reading difficulties, 

. School officials were asked to respond to a "School Prin- 
cipal's Questionnaire," which included questions about the 
size' and type of comtnunity served by the schools. In addi- 
tion, school officials in schools in which c it i zenshi p/ so- 
cial studies exercised' were administered were asked to 
respond to an "Instructional Program Questionnaire," which 
asked about citi zeriship/ social studies programs in the 
school. Schools in which only mathematics booklets were 
administered did not receive an " In stride tional Program Ques- 
tionnaire", Students also provided family or personal back- 
^^round information through questions included in the exer- 
' cise- booklets. Copies of forms used to collect background 
information from students and school officials in the 
1981-82 mathematics and citizenship/ social studies assess- 
ments are included in Appendix 1, 
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The assessment administrator coded for each respondent on 
birth date, sex, grade and racial/ ethnic classification on 
his or her booklet^ Administrators made a visual racial/ 
ethnic identification at the time each booklet was turned 
in. For the 1981-82 assessments, six different racial clas- 
sifications were used: American Indian or Alaskan Native, 
Asian or Pacific Islander, black, Hispanic heritage, white, 
and other or unclassified. If an administrator was unsure 
of a respondent's racial/ethnic group, he or she referred to 
the respondent's name or listened ^to the respondent's r^^ech 
pattern to make the identification. Respondents wei Jot 
verbally asked to give a racial identification for them- 
selves by the assessment administrator; however, respondents 
were asked to provide this information in one of the back- 
ground questions included in the assessment booklet at ages 
13 and 17. 

Following data collection, assessment administrators sent 
completed materials to the scoring contractor, Westinghouse 
Information Services (formerly Measurement Research Center) , 
Iowa City, Iowa. Booklets were quality-checked to verify 
that correct administrative procedures were being followed 
by the field staff and that all materials were accounted 
for. Coded identi fication ' information was also checked .or 
accuracy. Inconsistencies that could- not be reconciled were 
sent baek to the assessment'; administrator to be checked 
against the list of student names and identification nunbers 
retained by the school for six months following the assess- 
ment . 

For detailed information about the 1981-82 National 
Assessment data collection procedures, see Final Report-Year 
13 In-School Field Operations and Data Collection Activi- 
ties, National Assessment of Educational Progress^ Research 
TViangle Park, N.C. : Research TV-iangle Institute, Nov., 
1982. 
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Scoring 



Scoring and computer recording of data vwre contracted to 
Westinghouse Information Services (WIS), Iowa City, Iowa, 
for the 1981-82 mathematics and citizenship/ social studies 
assessments* National Assessment has found it most effi-> 
cient to have scoring done by an outside contractor. By 
having the same contractor do both the machine scoring and 
the open-ended or hand-scoring, booklets to be scored did 
not have to be shipped to another location when different 
scoring methods were needed . In addition , the scoring 
contractor has a trained staff of scoring personnel that can 
be called upon and augmented vAen , National Assessment 
conducts a major scoring effort. 

c ' 

Responses to " all multiple-choice exercises were read 
directly^ by optical scanning machines. The scoring 
contractor employed a special staff to hand score open-ended 
exercises, fecial training sessions were condxxsted by NAEP 
staff and WIS scoring administrators to insure that there 
was a high degree of uniformity in scoring procedures. 
Scorers were responsible for categorizing open-ended 
responses. They used scoring guides that defined discrip- 
tive categories of acceptable and unacceptable responses. 
They then coded this information into ovals that could be 
read by the optical scanning machine. 



TY^aining of Scorers 

Scoring guides for open-ended exercises were developed 
using field test data. Scoring categories included likely 
errors and thus were useful in identifying frequently made 
mistakes . . 

National Assessment staff worked with^ WIS staff to train 
scorers. Scorers were trained to use the scoring guides by 
scoring samples of responses taken from arriving assessment 
data. Scorers initially worked, as a group and discussed the 
appropriate categorization of each example response. They 
then worked individually on another^ set of responses. 
Discrepancies were resolved and explained. Once the group 
felt comfortable using the guides, .they started categorizing 
the actual data. Supervisory personnel checked all work 
done for the first few days of a scoring effort to be sure 
that scoring was consistent. 
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Scoring for each age group of mathematics exercises began 
during the administration of the assesatnent for- that age 
group, 'and it took from six to twelve weeks to canplete each 
age. TWo teams of eight scorers each- scored mathematics 
items. Oie team of ten scorers categorized the citizenship/ 
social studies responses. Qpen-ended citi zen ship/ social 
studies exercises were given only for ages 13 and 17^ 



Quality Control 

To further insure the quality and consistency of open- 
ended scoring quality-control checks were conducted. At 
regular intervals, randomly selected responses were drawn 
from the total pool of responses for an <e^)(;ercise and read by 
randomly selected scorers. Both the responses and the 
scorers Were selected without replacement, and approximately 
10% of the responses were included in the quality-control 
check. Scores for the quality-control readings were 
recorded, and the responses selected for quality control 
were then put back into the total pool of responses to be 
scored during the regular scoring the following' week. 
Following both scorings (quality control and regular), the 
two scores for quality-control responses were compared. 
When it seemed necessary because of poor agreements of 
regular and quality control scorings, scorers were retrained 
and, on some occasions, work was rescored. The average 
percentage of agreement between scorers during quality 
control checks for mathematics was 99? at each of the three 
ages; for citizenship/ social studies, it was 92% at age 13 
and 93% at age 17. 



Mathematics Scoring 

For changes in performance to be measired accurately, 
scoring procedures and guides .had to be the same for re- 
sponses collected in different assessment years. The same 
scoring guides and procedure^ that had been used in 1977-78 
were used again in 1981-82 to score open-ended exercises 
from the I977-78 mathematics assessment that were readminis- 
tered in 1981-82. When possible the same example responses 
used to train scorers for the 1977-78 assessment of mathe- 
matics were used to train scorers in 198V82. 
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Responses from the 1977-78 assessment to open-ended math- 
ematics exercises that were administered again in 1981-82 
were randomly selected from each primary sampling unit in 
the coiJntry, These 1977-78 responses were rescored in 
1981-82 to see if the 1981-82 scores were"^. consistent with 
the scores given in 1977-78. Since agreement between 
scoring done in 1977-78 and that done in 1981-82 was gener- 
ally 97? or more, NAEP concluded that scoring of the 1981-82 ^ 
responses would be consistent with scoring done in' '1977-78. 



Citizenship/Social Studies Scoring 

To insure the accuracy of change measures for the more 
complex open-ended citizen ship/ social studies exercises, 
responses from the 1975-76 assessment were rescored along 
with responses from the 1981-82 assessment. Scoring guides 
used in the 1975-76 assessment were revised and response 
examples from both assessment years were included to help 
define acceptable and unacceptable categories. Ihese 
revised guides were used to score both 1975-76 and .1981-82 
assessment responses. 



Exercise Level Analysis 

Ihe following description of NAEP data analysis proce- 
dures are presented as only one possible : set of procedures 
appropriate for NAEP data. The b'asic measure of achievement 
in National Assessment reports is the weighted percentage of 
respondents who^gave an acceptable response to a given exer- 
cise. 'This percentage is an estimate of the percentage of 
9-» 13- or 1 7-year^-olds who would have responded acceptably 
to a given exercise JLf every - 9-f 13- or 17-year-old in the „ 
country were asisessed. Since each exercise or exercise part 
is designed to measure some aspect of an objective or subob- 
jective, the percentage of correct or acceptable responses 
indicates an estimated level of performance for the partic- 
ular task at one point in time. ^ A comparison of the 
percentage of acceptable responses on" ±he same exercise for 
more than one point in time indicates whether proportions of 
the population able to perform the particular task are 
changing. 



Data Analysis 
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Procedures for estimating percentages of acceptable 
responses to mathematics and citizenship/ social studies 
exercises are dependent on the sample design. Each individ- 
ual's response is weighted and multiplied by an adjustment 
factor for nonresponse to reflect his or her selection prob-, 
ability. An ejjtimate of the percentage of a particular age 
group who would have responded to an exercise acceptably if 
the entire age group were assessed is defined as the 
Weighted nunber of acceptable responses divided by the 
weighted nunber of all the responses. A similar ratio of 
weights is used to ' estimate percentages of acceptable 
responses for reporting groups or subpopulations of 
interest. ' 

When performing trend analysis across assessments. . 
National Assessment, has explored a nunber of weight 
smoothing procedures'^ that are intended to reduce sampling 
variability in weights both between and within' assessments. 
A sunmary of several procedures and details on the approach 
currently recommended are contained in Appendix 2. 



Group Level Analysis 

In addition to providing the percentage of correct or 
acceptable "responses on each separate mathematics or citi- 
zenship/social studies exercise. National Assessment also 
reports the average performance across groups of similar 
exercises t such as the learning area as a whole, or a 
particular objective or subobjiective. For most assessments, 
the metric used for siwmarizing results is the mean of the 
estimates of performance on the group of exercises, and it 
i^* called the mean percentage correct or mean percentage of 
acceptable responses. In the early years of NAEP the median 
was used; however, an experimental study of change measures 
sponsored by NAEP indicated that the sampling stability of 
the mean change was as good as or better than that of the 
several other measures of central tendency examined. 

n .r 

National Assessment reports on the achievement of various 
subpopulations of interest as well as for the natioft as a 
whole. Groups are defined by region of the country, sex, 
race, size and type of community, level of parents* educa- 
tion and grade in school as well as by other characteris- 
tics. The difference between percentages of acceptable 
responses or averages for a reporting group and that of the 
entire age group (nation), on an exercise or groyp of exer- 
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cises can be used to describe the performance of any 
reporting group relative to the entire age group. See 
Appendix 3 for definitions of the standard NAEP background 
variables • 



Differences in Performance 

One of the most useful National Assessment achievement 
measure is differences in performance across time. Main- 
taining the same exercise* or sets of exercises in making 
these comparisons provides a reasonable indicator of whether 
more or fewer people know or can do something judged Impor- 
tant. To present a picture of changes in achievement for 
mathematics and citizenship/ social studies. National Assess- 
ment describes the gains or losses on an exercise or group 
of exercises in terms of the differences in percentages or 
mean percentages of correct, or acceptable responses. These 
differences may be used to see changes in achievement for 
the nation as a whole and for specific subpopulations . 



Precautions for Data Analysis 

- Unless the exercises sunmarized in 6he mean percentages 
of acceptable responses are identical » the means of one 
age group should not be compared to the means of 
another age group. 

- When only a few exercises are sx^nmarized by a mean, one 
should be especially cautious in interpreting results, 
since a small set of exercises might not adequately 
cover the whole range of potential behaviors included 
under a given objective or subobjective . 

- The mean should not be construed as an average test 
score, rather it should be interpreted literally as the 
arithmetic average of the percentage of acceptable 
responses obtained from National Assessment samples on 
a specific set of exercises. 
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Estimating Variability In Aohlevement Measures 

Katlonal Assessment uses a national probability sample at 
each age level to estimate the proportion of people v*io 
would successfully complete an exercise. The particular 
sample selected Is one of a large nunber of all possible 
samples of the same size that could have been selected vrith 
the same sample design. Since an achievement measure 
computed from each of the possible samples would differ from 
one sample to another, the standard error statistic Is used 
as a measure of the sampling variability among achievement 
measures from all possible samples. 

In addition to sampling variability, the standard error 
provides an estimate of other random error associated with 
the assessment. The standard error Includes all nonsystem- 
atlc error associated with administering specific exercises 
to specific students in specific situations. It also 
Includes random differences among scorers for open-ended 
exercises. , 

In the Interest pf sampling and cost efficiencies. 
National Assessment uses a complex, stratified, multistage 
probability sample design. Typlcairy, complex designs do 
not provide for unbiased. or simple ccxnputation of sampling 
errors. A reasonably good approximation of standard error 
estimates of acceptable response percentages is obtained by 
applying the jackknlfe procedure to first-stage sampling 
units, or replicates, within strata. Standard errors for 
achievement measures such as group differences, mean 
percentages or mean group differences for a particular 
assessment year are estimated directly, and they take advan- 
tage of features of the jackknlfe procedure that are generic 
to all of these statistics. Appendices 4 and 5 contain more 
details about National Assessment's approach' to estimating 
standard errors. 
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CXiallty Control 

The quality control for National Assessment is largely 
determined by the assessment design. The assessment is 
intended to measure changes in performance over time and to 
provide products and services for users, Eicercises must be 
sufficiently well-docunented to allow reuse by NAEP or 
external users. Thus, printing , sampling , data collection, 
scoring, analysis and data processing procedures must be 
replicable'at some time in the future. Also, the assessment 
must be able to detect average changes at the national level 
of one to two percent in the percentage of correct re- 
sponses. This precision requirement -places severe 
constraints on the amount of either sampling or nonsampling 
error that can be tolerated at any stage of the project. 
All of these factors require a much higher level of docunen- 
tation and quality control than is required in a small, 
one~time project. While quality control procedures exist 
throughout the assessment, the remainder of this section 
focuses on quality control in data collection, scoring, data 
processing and analysis. 
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Data Collection 

All data collection is performed by a specially trained 
field staff, Ihe quality of data collection is monitored 
through: 

- Field visits by NAEP and contractor supervisory staff . 

- Interviews of school staff and respondents, 

- Analysis 'of problems reported by the scoring 
contractor, 

- Analysis of quality control que^stionnaires received 
from, participating schools and districts. 



Scoring 

Scoring activities are divided into three major areas: 
receipt of data from field staff; hand scoring; and machine 
scoring and processing/ 

• Daring receipt oTdata from the field staff, all mater- 
ials are checked for: 

- Timely receipt of all materials distributed to field' 
staff. 

- Accuracy of demographic data, 

- Completeness. 

- Adherence to specified procedures, 

- Special problems requiring follow-up with field staff, 
the data collection contractor or NAEP, 

Daring hand scoring of open-ended exercises, quality 
consistency is insurei^ through the follQWlng procedures: 

- Training scorers. 

- Monitoring performance of scorers, 

- Determining* the consistency of scoring. 
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- Regular quality control readings of sample responses. 

In machine scoring and processing, the contractor main- 
tains a . -Scanner error rate of less than one per hundred 
thousand characters through engineering checks and the 
following procedures: 

- Tests by the contractor of all programs. 
Tests by NAEP of all programs using test data. 

- Manual editing of all illegal values in data fields. 



Data frocessing and Analysis 

To provide independent checks on data accuracy, a great 
deal of auditing is built ,into data processing and analysis 
procedures. Major quality control steps include: 

- Manually editing and machine editing all exercise and. 
other data file docunentation. 

- Manually scerifying respondent diata files against orig- 
inal contractor files. 

- Verifying weight distributions from NAEP files against 
contractor distributions and ccmparing with past data 
and census data. 

- Docunenting audit trails of all processing steps and' 
diagnostic data. 

- Computing preliminary and final data Independently to 
provide checks of results. - 

- Editing response ranges and consistency of data fields 
during contractor processing, NAEP respondent file 
loading, preliminary analysis and sunmary file loading. 

- Verifying control totals for all analyses. 

- Checking unusual or inconsistent patterns of results. 
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Background Variables 

National Assessment collects background information in 
addition to responses to assessment exercises. Typically, 
exercise variables are cognitive, attitudinal or experien- 
tial questions that relate to learning area objectives. 
Background variables are all other information about respon- 
dents, their schools or communities, data collection condi- 
tions, etc. It is not always clear vyhether a particular 
variable ougjj^ (in some sense) to be a background or exer- 
cise variable. 

Background information is collected from a number of 
sources. Some information comes from the exercise booklets, 
and some is provided by school officials responding to ques- 
tionnaires. Some information is collected from observations 
by the test administrators, from school records or from 
sampling records. 

Some background variables are recoded or derived from the 
original information gathered and are called derived vari- 
ables. For exanple, parental education is derived trom 
father's education and mother's education. 

Some background information is collected about each indi- 
vidual respondent, and some is collected for each partici- 
pating school or for some larger level unit. 

NAEP background variables may be divided into three 
categories: 1) standard NAEP background variables, 2) other 
common NAEP background variables, and 3) year-specific NAEP 
background variables. Ihe following describes these three 
types of background variables. 



Standard NAEP Background Variables 

IWelve standard background variables have a consistent 
definition across ages and assessments. These variables are 
prisent for almost all assessment years. These variables 
are: 

Census Division (9 divisions) 

&*ades in School (collapsed into most 

common categories) 
Race/ ethnicity (3 categories) ' 
Race/ ethnicity (4 categories) 
(immunity Size 
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HDme Ehvironment 
Modal G^ade 
Parents' Education 
Geographio Peg ion (4 regions) 
Sex 

Size and Type of Community 
Type of Community 

Definitions for most ^standard variables can be found in 
Appendix 3t the CLossery. 

In some cases standard NAEP variables are defined 
directly from the background information that has been 
collected. In other cases these variables are derived from 
other variables. Across years, the variables may be based 
on different category systems. For exanple, racial/ethnic 
categories have evolved over time from the simple black/ 
, white/other categories used in 1969-70 to the current Amer- 
, ican Indian or Alaskan Native/ Asian or Pacific Islander/ 
black/ Hispanic heritage/white/ other categories, tovgever, 
by recoding datat the variables are made consi3ten^_^cross 
all years. The four category (black, white, Hispanic 
heritage, other)racial/ethnic. variable provides a consistent 
variable from 1972-73 to the present, while the three 
category (black, white, other) racial/ethnic variable 
provides a consistent racial/ ethnic variable for all assess- 
ments except 1970-71. In that year only, Hispanic respon- 
dents were coded as vAiites; In all -other years Hispanic 
s respondents were coded as other. 



Year-Specific KAEP Background Variables 

For each assessment, background information is collected 
relevant to the particular content area being assessed. For 
the most part, these data are collected from students in the 
form of ' specific background questions. These are either 
included in most or all exercise booklets or asked of school 
personnel through an instructional program questionnaire. 

School background information on the following topics was 
colleted in 1981-82: 

- Social studies teaching staff 

- Social studies materials available in the school 
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- Type of materials and/or teaching approaches used to 
teach citizenship/ social studies. 

- Enrichment activities used in teaching citizenship/ 
social studies 

- Instruction given to enable student to obtain skills 
and knowledge in 

A. Acquiring information 

B. Using information 

C. Understanding individual development 

D. Understanding the ways hunan beings organize 

E. Understanding the development of the Ihited States, 

No school-level instructional program questionnaire about 
mathematics instruction was given in 1981-82. Students 
responded to questions about the kinds of mathematics-re- 
lated experiences that they had had. Responses to these 
items were- used to create background variables dealing with 
use of the metric system, use of hand-held electronic calcu- 
lators and, for age 17, the type of mathematics course^^ 
taken. 



Data Origins , - 

The data used by National Assessment in their analyses 
come from a variety of sources. The following describes 
these sources of information. ^5any of the forms used are 
found in Appendix 1. 



Observation by Exercise Booklet Administrator 

Respondent-level sex and race identification and adminis- 
tration information are provided by the field administration 
staff as they conduct exercise booklet administrations. 
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Respondent Questionnaire 

Respondent-level background questions are inclvxied at the 
end of every exercise booklet (see Appendix 1). 



Respondent Data 

Respondent-level exercise data comprise the major portion 
of exercise booklets. 



School Principal's Questionnaire 

A standard school-level questionnaire is used for 
updating sampling records, constructing the size-and-type of 
community variable and providing background data (see 
Appendix 1). When possible, missing data are estimated from 
secondary sources, including Census, Office for Civil 
Rights, state directories and (Xirriculun Information Center 
files. 



Instructional Program Questionnaire 

At all three ages, a special instructional program ques- 
tionnaire was administered to all sample schools that 
received booklets of citi zenship/ social studies exercises 
(see i^pendix 1). Ihe questionnaire asked about instruc- 
tional methods and materials related to citi zenship/ social 
studies instruction. 



School Records 

Besides respondent's birth month and year and grade in 
school, school-level data are obtained , from school personnel 
or records. 
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Sampling Records 



Sampling records are used' for a variety of^ldentlfiers, 
demographic variables and/" other data provided by the 
contractors from a nimber of sources. Except for respondent 
identification nunbers, all variables are school-level or 
higher (region, community size, etc,)-. 



Derived E&ta 

Some variables are constructed or derived from or*iginal 
data fields by National Assessment staff. 
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CONTENTS . ' 

Background Variabl e Ques t ions 

AGE PAGE 

1. Citizenship/Social Studies Background 

Questions. ..... . 13,17 2 

2. Citizenship/Social Studies Background 

Questions . .... . . .* 13,17 3 

3^. Citizenship/Social Studies Background 

Questions. .... 9,13,17 4 

4. Mathematics Background Questions 9 5 

5. Mathematics Background Questions 13,17 6 

6. Mathematics Background Questions ...... 17 '7 

7. Mathematics Background Quest ions - 17 8 

8. Citizenship/Social Studies and Mathematics 

Background Questions 13,17 9 

9. Citizenship/Social Studies and Mathematics 

Background Quest ions 13,17 10-11 

10. Citizenship/Social Studies and Mathematics 

Background Questions 13,17 

U. Citizenship/Social Studies and Mathematics 

Background Questions * . . 9 14, 

12. Citizenship/Social Studies and Mathematics 

Background Questions 13 15 

13. Citizenship/Social Studies and Mathematics 

Background Questions 17 16' 

14. Citizenship/Social Studies Instructional Program 

Questionnaire. ................ 9 17-24 

15. Citizenship/Social Studies Instructional Program 

Questionnaire. 13,17 25-32 
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Have you ever studied how to use any of the following? Fill in only one oval 
for each part. 



Yes No I do.n*t know, ^ 



A. 


Maps czD 


. — — " — I 


B. 


Globes czD 


i 

C3 CZD . j 


c. 


Charts czD 


dD CZD 


D. 


Tables c=D 


dD CZD 


E- 


Graphs czD 


CZD CZD 



CI3 



CD 




DO NOT OONTlNL'b; 

I NTIL rOLD TO DO SO. 
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A. How often do you talk about controversial topics to your. TEACHERS? 



Hardly ever 
One day a week 
Two days a week" 
Three or four days a week 
CZD More than four days a week 



B. How often do you talk about controversial topics to your PARENTS? 



Hardly ever 

One day a week 

Two days a week 

Three or four days a week 

More than four days a week 



C. How often do you talk..about controversial topics to your FRIENDS? 



Hardly ever 

One day a week 

Two days a week 

Three or four days a week 

More than four days a week 



1 m 


CID 


i * 


CID 




OD 


2 


CO 


1 m 


CO 


6 m 


CO 


8 - 


CO 




CO 


2 


CO 


1 m ' ' 


CO 


O 
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STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



C2D 
CD 
CO 
CO 
CZD 
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How often do you talk about government or politics to your 

TEACHERS? ^ i 



Hardly ever 

One day a week 

Two days a week 

Three or four days a week 

More than four days a week 



B. How often do you talk about government or politicslto your PARENTS? - 



Hardly ever 

One day a week 

Two^days a week 

Three or four days a week 

More than four days a week 



C. How often do you talk about government or politics to your FRIENDS' 



Hardly ever 

One day a week 

Two days a week 

Three or four days a week 

More than four days a week 



l^^^r^l DO NOT CONTIM'E 
^ STOP UNTIL TOLD TO DO SO. 



ERLC 



CZD 
CID 

».iHHHH»iAI»ll»*l.2.: 
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For each of the following questions; fill in one oval in each box. 



The metric system uses units like centimeters, liters and kilograms. 
Have you used the metric system of measurement? 



Yes 



No 



I don*t know. 



B. 


How often have you used the metric system in 


mathematics? 




Often Sometimes Never 

CZD CD CZD 


' I don't know, 

CD 


C. 


How often have you used a hand calculator? 






Often Sometimes Never 


I don't know. 

CD 


D. 


Do you or your family own a hand calculator? 
Yes - No I don>know. 

CID CD CD 





E. 



Does your school have hjind calculators that you can use in 
mathematics class? 



Yes 



No 



I don't know. 



CD 
CD 
CID 
CD 



CD 
CD 



00 NOT CONTINUE 

L NTIL TOLO TO DO/ SO. 



4D 
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For each of the following questions, fill in one oval in each box. 



A. 



The metric system of measurement uses units like centimeters, liters, 
and kilograms. How often have you used the metric system? 



Often Seldom 



Never I don't know. 



B. How often do you use a hand calculator? \ 

Almost A few Less than Once I don't 

Daily times a week once a week a month Never know. 



Does your school provide hand calculators for use in mathematics 
class? . 

Yes No I don't know. 











2 « 




1 


CO 


i m 


CO 


4 


CO 


2 m 


CO 


1 m 


CO 


6 m 


CO 


a 


CO 




CO 


2 m 


CO 


1 


CO 


o 





.v»Mu»im:,K!i| I 



DO NOT CONTINUE 
LNTIL TOLD TO DO SO. 
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For each of the following questions, fill -in on.e oval in.each box. 



The metric system of measurement uses units like centimeters, liters, 

and kilograms. How often have you used the metric system of 

measurement? 



Often 



Seldom 



Never 



I don't know. 



B. How often do you use a hand calculator? 

Almost A few Less than Once 

Daily times a week once a week a month Never 



I don't 
know. 



C. Does your schv ' ''^vide hand calculators for use in mathematics 
classes? 



Yes 



No 



I don't know* 



D. Does your school provide hand calculators for use in other classes? 
Yes No ^ I don't know. 



5i 




DO NOT CONTiNXl- 
UNTIL TOLD TO DO SO. 
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Which of the following mathemntics courses have you studied? Fill in one oval 
on each line. ,(If you have not studied a particular course, fill in theoval under 
"Not Studied". )^ * . 



Studied 1 
school year 



Studied V2 
school year 



Studied' I 
less than Not don't 
^ year studied know. 



A. 


General, Business 
or Consumer 
Mathematics 




CD 


CD 


CD 


CZD 


B. 


' Introduction ' 

to Algebra 
. (Pre-Algebra) 


CD 


CD 


CD 


CD 


CD 


C. 


1st year- 
Algebra ^ 


C=D 


CD 

r 


CD 


CD 


. CD 


D. 


2nd year 
Algebra 


CD 


CD 


CD 


CD 




E. 


Geometry 


CD 




CD 


CD 


^ — 
CD\ 


F. 


Trigonometry 


CD 


CD. . 


CD 


" CZD 


CD^ 


G. 


Probability .& 
Statistics 


CD 


- CD . 


CD 


CD 


' CD ^ 


H\ 


Computer 
Programming ' 


CD 


CD 


CD 


CD 


CD 


I. 


Pre-Calculus/ 
Calculus 


CD 


CD 


CD 


CD 


CD 



CD 
CD 
CO 
CZD 
GO 



CZD 



52 




DO NOT CONTINUIZ 
UNTIL TOLD TO DO SO. 
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How much television did you watch yesterday? 



cz) None 

cz) 1 hour or less 

CZ) 1 hour 



CZ) 2 hours 
CZ) 3 hours 
CZ) 4 hours 



CZ) 5 hours 

CD 6 hours or more 



Is English the language spoken most often in your home? 

CZ) Yes 
CZ) No 

Is a language other than English spoken in your home? 

CZ) Often ^ 
CZ) Sometimes 
CZ) Never 



28 

C2D 
GO 
CO 
CO 
CO 
CO 
OD 
CO 
CO 
CO 



29 

CO 
CO 
CO 
CO 
CO 
CO 
CO 
CO 
CO 
CO 
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STOP 



DO NOT CONTINUE 
UNTIL TOLD to do SO, 
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Which ni tlui following does your family have at home? (Fill in one oval on 
each line.) 



Have 



Do not have 



A. N'.'wspuptT ret-eived regularly c=5 ^ 


B. 


MuKiu'.iiH's received regularly 


CD CD 


C. 


More tluui 25 books 


CZ) CD 


. D, 


KiK'yi'l()i)odia 


CD CD 


1 _. . . 

E, Piotionary ^ ^ 


r — 

, F. 


Record player 


CD CD ■ 


i 


A hand held calculator 


CD CD 


H. 


A (M)mputer (NOT a hand held calculator) 


CD CD 


I. 

i 


Ty pewriter 


CD CD ^ 


.: J. 


Vacuum cleaner 


CD CD 


r 

! ^- 

i 


Electric dishwasher 


CD CD 



10 

1 



ERIC 



How much television did you watch yesterday? 
Xone ^ 2 hours CZ) 5 hours 

1 hour or less <=> 3 hours <=> 6 hours or more 
1 hour ^ CZD 4 hours 



A How manv brothers or sisters do you have who are older than you? 

None' czDl <=> 2 (=> 3- c=d4 c=d5 c=)6or more 



B How marv brothers or sisters do you have who are vgunger than you? 

None^ CD I c=> 2 c=) 3 c=) 4 c=) 5 c=) 6 or more 



White ■ 

Other (Please Specify) 



A. What is your racial background? 

American Indian or Alaskan Native 
Asian- or Pacific Islander 

Black 



\y 

B. !s your ethnic heritage Hispanic? 
Xo (Not Hispanic) 

Yes (Mexican, Mexican-American, Chicano) 
Vl'S. Puerto Rican 

Yes. Cuban ^ . , 

Ves. other Spanish/Hispanic 

A. Is tiVcTiish the language spoken most often in your home? 

^ Yes CID No , / 

t 

B. I/, :i ;u;v^uaire other than English spoken in youi;: home? 
r — (if-en c=3 Sometimes c=D Never 
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How ofUMi has each of the following been used in the courses you are taking 
this year? t Fill in one oval on each line.) 







Never 


Seldom 


Fairly 
Often 


Frequently 


: A. 


Li.-^tenin^ to the teacher's lecture 






CZ) 


CZ) 


■ B. 


Participating in student-centered 

diseussions 






CZ) 


CD 


C. 


^Vorking" on a project or in a 
laboratory 




CZ) 


CZ) 


, CD 


i ^- 

i 
1 


Writing essays, themesv poetry, 

stories 


CZ) 


CZD 


CZ) 


CD 


: 


(ioinir on field trips 




CID 


CZ) 


CD 


1 F. 

I 

1 
i 


Having individualized instruction 
(small groups or one-to-one with 
a teacher) 


C=3 




CZ/ 


CD 


i G. 


Usin^ teaching machines or 
com puter-assisted instruction 


CZD 


f — 


CZ) 


CD 




Watching t( levision lectures 


CZD 


CZD 


CZ 


CD 


i [. 

i 


Studying from textbooks 


CID 


CID 


CZ) 


CD 


1 J. 

1 


Librar\' or media center assignments 


C=D 


c — ) 


CZ) 


CZ) 



How much lime did you spend on homework yesterday? 

CZ) No homework was assigned. 

CZ) I had homework but d idn't do it. . 

CZ) Less than one Hour 

CZ) Between 1 and 2 hours 

CZ) More th:in 2 hours 



Which one of the following best describes your grades so far in school? 



CZD Mostly A 






CID About ha 


f A and half B 




CID Mo.ily B 




CD 


CZD ,\i)o;.;t ha 


f B and half C ■ ' 


CD 


C=D Mostiy r 








f I - and half D 


gd' 


^ ^ ^ ' , I ■ j • • [ ) 


!(;'.V D 


CID 




CZD 






CO 






(ZD 
CID 









Do 
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SU. Which of ihv I'ollowin^^ doe.^ your family have at home? (Fill in one oval on 
each line.) 



iEST COPY Al/AiAl 



SLE 



12 
1 







Have 


Do not have 


— • 


A. 


• Newspaper received re^^ularly 


CZD 


CZD 




B. 


Magazines received regularly 


CZD 


CZD 




c,. 


More than 25 books 


CZD 


CZD 




D. 


Encyclopedia 


CZD 


CZD 




E. 


Dictionary 


CZD 


CZD 




! F. 

1 


Record player 


CZD 


CZD 


— • 


G. 


A hand held calculator 


CZD 


. CZD 




H. 


A computer (NOT a hand held calculator) 


CZD 


CZD • 




I. 


Typewriter 


CZD 


^CZD 




J. 


Vacuum cleaner 


' CZD 


C=) 




K. 


Electric dishwasher 


CZD 


CZD 


III 



How much television did you watch yesterday? 
None CZD 2 hours CID 5 hours 

1 hour or less czD :i hours CID 6 hours or more 
1 hour CZD 4 hours 



81. 



82. A, How many brothers or sisters do you have who are older than you? 



None 



1 



4 



6 or more 



B, How many brothers or sisters do you have who are younger than you? 

None' cz:^ 1 czd 2 czd 3 czd 4 czd 5 czd 6 or more 



83, A. What i^^-our racial back.irrou nd? 

CZD' Arperican Indian or Alaskan Native 
CZD x\silrn or Pacific Islander 
Black 



White 

Other (Please Specify) 



B. Is your ethnic heritage Hispanic? 
No ( Not Hispanic) 

Yes { Mexicap.. Mexican- American. Chicano) 
Yes, Pucrtf), Ilican 
Yes. Cuban 

Yes. olhor S panis!^. Hispanic 



84. A, Is Enjrlish ilv.- la r-i-ua.cc spoken most oi'icn in your home. 
Yes No 



.CO 



B. Is a laniriia^jfc h'.h^'V Kn.irlish spoken in your home? 

cz? Ofti/n .~' Sometimes' czd Xever 



EKLC 



5g 



irt' 



l.:F!Mt 



en) 



BEST copy AVAILABLE 
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How often has dach of the following been used in the courses you are takintn: 
this year? (Fill in one oval on each line.) \^ 







Never 


, Seldom 


Fairly 
Often 


Frequently 


A. 


' Listening to tlie teacher's lecture 






CZD 


CZD 1 


D 
D . 


r a 1 11 c 1 pa 11 n in oLuucnt'LcnLcrcu 
discussions 


CZD 


CUD 




c — J i 

CZD 1 

1 


C. 


Working on a project or in a 
labaratory 


C=D 




CZD 


i 


D. 


Writing essays, thennes, poetry, 
stories 




C=D 




CZD 1 

i 

I 


h . 


Going on field trips 


C ) 


C ) 


CZD 


CZD 


b . 


Having individualized instruction 
(snriall groups or one-to-one with; 
a leac ner } 


C=D 


CZD 


CZD 


t 

1 

i 


G. 


Using teaching nnachines or computer- 
assisted instruction 


(ZD 




CZD 


CZD 


H. 


Watching television lectures 


C=D 


CZD 


CZD 


CZD 


I. 


Studying from textbooks 


C=D 


CZD 


CZD 


CZD 


J. 


Library or media center assignments 


C=D 


. ..-CZD 


CZD 


-GZD T 



How much time did you spend on homework yesterday? 

CZD No homework was assigned. 

CZD I had homework but didn't do it. 

CZD Less than one hour 

CZD Between 1 and 2 hours 

CZD More than 2 hours 

Which one of the following best describes your grades so far in high school? 

CZD Mostly A 

CZD About half A and half B ' 

CZD M.ostlv B 

czd' About half B and half C 

CZD Mostly C 

CZD About half C and half D 

CD Mostly D 

CZD Mostly below D r r^ 

CZD 

Which one «)f the following best describes your present hii^h scht)oi pr(.)j:rram? czD 

CZD 

CZD (lun oral CZ3 
CZD Academic or college preparatory czD 
CZD \'()eational or technical CED 
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Does 



BEST COPY AWMKIF 

your ramily ^^et a newspaper resvilaily?*' »^A3fiUralL5L£ 



dD Yes (ZD No CZD I don't know. 
Does your family get any niaj^azines regularly? 
dD Yes . CID No cz^ I don't know. 
Are there more than 25 books in your home? 
CZD Yes CZD No CZD I don't know. 
Is there an encyclopedia in your home? 
( — ) Yes ( — ) No CZD I donx know; 
How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 

CZD Did not complete the 8th grade 

CZD Completed the 8th grade, but did not go to high school 

CZD Went to high school, but did not graduate from high school 

CZD Graduated from high school * 

CZD Some education after graduation from high school 

CZD I don't know. 
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Did your father graduate from a college or university? 
CZD Y'es CZD No CZZ) I don't know. 



How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
mother completed.) 



Did not complete the 8th grade 

Completed the 8th grade, but did not go to high school 
Went to high school, but did not graduate from high school 
Graduated from high school 
Some education after graduation from high school 



Xdonlt-know. 



n : your mother graduate from a college university? 
CZD Yes CZD No CZD I dont know.. 



CZD 



ERIC 
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How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 

az) Did not complete the 8th grade 

t — ) Completed the 8th grade, but did not go to high school 

t — ) Went to high school, but did no^: graduate from high school 

t — ) Graduated from high school / 

t — ) Some education after graduation from high school 

CZ) I don't know. j 

Did your father graduate from a college or university? 
r— ) Yes dZ) No dZ) I don't know. 
How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 

mother completed.) 

<ZZ) Did not complete the 8th grade 

t — ) Completed the 8th grade, but did not go to high school 

( — ) Went to high school, but did not graduate from high school 

az) Graduated from high school 

( — ) Some education after graduation from high school 
az) I don't know. 

Did your mother graduate from a college or university? 
( — ) Yes az) No I don't know. 

Wh"eTe~^did you live on . ~" 

■ ( — ) In the United States (Please specify the state or territory.) 



t — ) Outside the United States (Please specify the^untry.) 



dZ) I don^t know. 



So 



ERIC 
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.How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 
CZ) Did not complete the 8th grade 

Completed the 8th grade, but did not go to high school 
Went to high school, but did not graduate from high school 
Graduated from hi gh so bool 
Some education after graduation from high school 
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CZ) I don't know. 

Did your father graduate from-a college-or- university? 
CZ) Yes CZ) No cz I don't know. 



How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows hpw much school your 
mother completed.) 
CZ) Did not complete the 8th grade 

CZ) Completed the 8th grade, but did not go to high school 
CZ) Went to high school, but did not graduate from high school 
CZ) Graduated from high school 

CZ) Some education after graduation from, high school 
CZ) I don't know. - 

Did your mother graduate from a college or university? 

(ZD Yes rszj No (ZD I don't know. 
- Where- did you J Lye on your .ninth, b irthday ?__ _ ^ _ 

(ZD In the United States (Please specify the state or territory.) 



Outside of the United States (Please specify the country.) 



I don't know. 
Where did you live on your thirteenth birthday? 

In the United States (Please specify the state or territory.) 

Outside of the Uirfted States (Please specify the country.) 

I don't know. ~" 7 



DO NOT 

WKixr: IN 

THE AUr A 
BKLt>W 

M I. J 



) 




BL czD cr.: 



Go 
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Year 13 
A^e Class 1 



NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 

Instructional Program Questionnaire 
i 

Citizenship/Social Studies 



The purpose of this questionnaire is to provide additional information which 
will be used in the analysis of the citizenship/social studies data. This 
questionnaire should be completed by the person(s) MOST FAMILIAR with 
the citizenship/social studies program in your school. If you have questions 
about any of the following items, please" contact the National Assessment 
District Supervisor. Thank you for your cooperation. 

Please answer the following questions regarding the citizenship/social studies 
program in your school. The term citizenship/social studies is meant to include 
any instruction or activity that relates to the study of human beings and their 
relationships with the social and physical environments as well as the study of 
the civic behavior of individuals and groups. Social studies would include any 
courses that involve concepts from the social science d isciplines of anthropology, 
economics, geography, history, philosophy, political science, psychology and 
sociology. 



PLEASE USE A SOFT LEAD PENCIL. 



1. Ts there a person available for the 
supervision and coordination of the 
social studies program in your school? 

2. Does your school provide release time 
for the social studies teachers to 
attend meetings or workshops for social 
studies professionals? 

3. Are the social studies teachers in your 
school involved in the textbook selection 
process? 



O M B No. 1850-0083 
Expiration Dale 09 82 




Does the copyright date on the average 
social studies or social science textbook 
used in vour school bear a date earlier 
than 1978? 



Appendix 

Yes ~ No 



Wi^thin the past three years, have funds 
been available in your school for 
s^Lipp 1 e m eMa. r V ,s^^^^ 
science materials and textbooks? 



Are any of the following materials and/or teaching approaches used in 
teaching citizenship/social studies in your school? 



Yes 



No 



A. Audio-Visual materials (e.g., tapes, films, 
filmstrips, maps, charts, graphs, tables) 



B. High Jnterest/Low Vocabulary Social Studies 
materials (e.g., workbooks/graded texts, 
study guides) 



C. Games and kits (commercial or teacher made) 



D. Instructional television 



E. Team teaching, contract teaching, learning 
centers, peer tutoring, learning packets, etc. 
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Are any of the followingeFnchm^nTa^ 
social studies to help students in your school? 



Yes No 



A. 


Field trips 




CZID 


B, 


Resource and/or community people 


CZ3 


CZ3 


C. 


Special projects (e.g., creative projects, 
exhibits, travel logs, school/community 
partnerships) 




'CZ3 


D, 


Dramatic activities (e,g,, plays, 
reenactments, stage construction, 
costume designing, puppeteering) 


CZ3 


CZ3 


E. 


Mock elections'^ 


CZ3 


CZ3 


F. 


Mock political conventions 


CZ3 


CZ3 



Does. your. schQX).L^islri.c.t.hax£-a_^^ 

of controversial issues in school? 

(ZD Yes czz) No , ' 

What percentage of your teachers are teaching ONLY social studies? 

ciD None 

CZ3 l-24?/o 

CZ3 25-49%^ 

CZ3 50-74% 

CZD 75-100% 
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During the past three years, what PERCENTAGE of your ENTIREteaching 
staff (including teachers in areas such as art, reading, music, mathematics, 
science, physical education, etc.) have participated in PLANNING inservice 
activities for your school or district? 

CZD None participated in planning 
cn) 1-9% participated in planning 
CZD 10-24% participated in planning 
CZD 25-49% participated in planning 
CZD 50-99% particj^pated in planning 
CZD 100% participated in planning 

During the past three years, what PERCENTAGE of your ENTIRE teaching 
staff (including teachers in areas such as art, reading/ music, mathematics, 
science, physical education, etc.) have received inservice TRAINING in 
CITIZENSHIP/SOCIAL STUDIES from your school or district? 

CZD No citizenship/social studies inservice training offered 

CZD None received training 

CZD 1-9% received training 

CZD 10-24% received training 

CZD 25-49% received training ... ' 

CZ) 50-99% received training 

CZD 100% receWed training 
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2. For each of the following areas, indicate whether or not th,e students in 
your school are given instruction in that area or are provided with 
activities that would help them develop knowledge and skills in'that area. 

A. ACQUIRING INFORMATION^. 







^ A part of 
instruction 
. or activity 


NOT a part 
of instruction 
or activity 


1. 


Use of the senses to 
obtain information 


CZD 


CZD 


2. 


Use of a variety of sources 
to obtain information 


CID 


CZD 


3. 


• 

Use of a variety of 
techniques (e.g., 
interviews, polls, surveys) 


CD 


' " • CZD 


B. 


USING INFORM,ATIOn' 








- 


A part of 
instruction 
or activity 


NOT a part . 
of instruction 
or activity 


1. 


Organizing information 

(e er. arranerincr errounincr 

relating, analyzing) 


CD 


CZD 


2. 


Applying information (e,g., 
inferring, predicting*, 
testing alternatives) 


CD 


CZD 


3. 


Decision making and 
problem solving 


CD 


CZD 


4. 


Critically evaluating 
information 


CD 


CZD 
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(Continued) 

C. UNDERSTANDING INDIVIDUAL DEVELOPMENT 





* 


A part of 
instruction 
or activity 


NOT a part 
of instruction 
or activity 


1. 


Examining the > 
individuaTs beliefs, 
values and behaviors 


CD 


! 

CD 


2. 


Individual development 
(setting- goals, determining 
consequences of goals) . 


CD ' 


CD 


3. 


Graphic and oral 
communication 


CD 


CD 


4 


Interfrersonal communication 


CD 


CD 


5. 


Interaction in groups 


CD 


CD 


' 6. • " 


' ^'R e t ati n g" t o~ p e6 pi e" of " ■ 
different cultural 
perspectives 


-CD 


CD 


D. 


UNDERSTANDING THE WAYS HUMAN BEINGS ORGANIZE 






A part of 
instruction 
or activity 


NOT a part ' 
of instruction 
or activity^f 


I. 


Ways in which people are 
interrelated (e.g.. 
economically, environmentally, 
.historically, politically, 
culturally, etc.) 


CD 


CD 


2. 


()r.franization.< of human 
societies (e.g.. nature oL 
institutions and groups!-^ 




1 

^ 
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12. (ContiruRMl) 

D. UN/DERSTANDIN&THEWaYS HUMAM BEIMGSORaANiZE 



(Conti nued) 



A part of 
instruction 
or activit\- 



NOT a part 
of instruction 
or activity 



Relationships l)et\veGn 
irul ivi(juals and Ki'oups 
(o.^".. roles in groups, 
effect of groups on 
ind iviciuals) 



Relationships between 
^?r:oups (e.R., cooperation 
and conflict amonp: groups) 



5. Global concei'ns [Q.g.. 

food, popuhition. disease, CZ3 CZD 

racism, conflict, human 

I'ights) 



6. Human rights worldwide 
(e.^.. acceptance and 
appreciation of universal cz3 
differences. e(]uitaJ)Ie 
treatment; consicleration 
^ for interest of others) 



E, UNDERSTANDING THE DEVELOPMilNT OF THE UNITED 
STATES 



c 


/ 


A part of 
instruction 
or activity 


NOT a part 
of instruction 
. or activity 


I m 
1 m 


1. 


Principles and purposes 
of the u.S- government 






i 


2. Organization and operations 
of U,S. government (federal, 
rtatei, local) j 


; 


I 

1 

CD 


1 










THANK YOU F^pFtYOUR COOPERATION^ 
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Year 13 o.m.b. no isso-ooaa 

Age Classes 2 and 3 Expiration Date 09/82 



NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 

Instructional Program Questionnaire 
Citizenship/Social Studies 



The purpose of this questionnaire i,s to provide additional information which 
will be used in the analysis of the citizenship/social studies data. This 
questionnaire should be completed by the person(s) MOST FAMILIAR with 
the citizenship/social studies program in your school. If you have questions 
about any of the following items, please contact the National Assessment 
District Supervisor. Thank you for your cooperation. 

Please answer the following questions regarding the citizenship/social studies 
program in your school. The term citizenship/social studies is meant to include 
any instruction or activity that relates to the study of human beings and their 
relationships,with the social and physical environments as welj as the study of 
the civic behavior of individuals and groups. Social studies would include any 
courses that involve concepts from the social science disciplines of anthrop'^Iogy. 
economics, geography, history, philosophy, political science, psycholo^ ' i 
sociology. 



PLEASE USE A SOFT LEAD PENCIL 



1. Is there a person available for the No 
supervision and coordination of the w-.^ CD 
social studies program in your school? 

2. Does your school provide release time Yes No 
for the socivvJ studies teachers to . czD fZZ) 
attend meet ings or workshops for social 

studies professionals? 

3. Are the social studies teachers in your Yes No 
school involved in the textbook selection C3 CZ) 
process? 
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Does the copyright date on the average Yes No 

social studies or social science textbook ctd ci3' 

\ used in your school bear a date earlier 
than 1978? 



Within the past three years, have funds Yes ' No 

been available in your school for (=3 CID 

supplementary social studies or social 
science materials and textbooks? 



Are any of the following materials and/or'teaching approaches used in 
teaching citizenship/social studies in your school? 



Yes 1^0 



A. 


Audio-Visual materials (e.g., tapes, films, CZD 
filmstrips, maps, charts, graphs, tables) 


(=3 


B. 


High Interest/Low Vocabulary Social Studies 
materials (e.g.. workbooks, graded texts, CZD 
study guides) 


CD 


C. 


Games and kits (commercial or teacher made) , ctd 


CID 


D. 


Instructional television ctd 


CID 


E. 


r -run U'v. h'ng ^.;ontract teaching, learni ng cn) 
c .'u'r:.;, .^^..'r ^ atfjring. learning packets, etc. 


CID 



I 



I 
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Are any of the following enrichment activities used in teaching citizenship/ 
social studies to help students in your school? 







1 es 


Mn 
iS 0 


A. 


Field trips 


CZD 


CZD 


B. 


Resource and/or community people 


cz> 


CZD 


C. 


Special projects (e.g., creative projects, 
exhibits, travel logs, school/community 
partnerships) 


CZD 


CZD 


D. 


Dramatic activities (e.g., plays, 
reenactments, stage construction, 
costume designing, puppeteering) 


CZD 


CZD 


E. 


Mock elections 


CZD 


CZD 


F. 


Moqk political conventions 


CZD 


CZD 



Does your school district have a written policy which allows the discussion 
of controversial issues in school? 

CZD Yes CZD No 



What is the average number of years of teaching experience for the social 
studies teachers in your school? 

CZD Less than 1 year 

CZD 1-2 years 

CZD 2-3 years 

CZD 4-5 years - 

CZD More than 5 years 
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10. What percentage of your social studies teachers are certified to teach 
social studies? 



None 

1-24% 

25-49% 

50-74% 

75-100% 



11. During the past three years, what PERCENTAGE of your ENTIRE teaching 
staff (including teachers in areas such as art, reading, music, mathematics, _ 
science, physical education, etc.) have participated in PLANNING inservice 
activities for your school or district? 



None participated in planning 
1-9% participated in planning 
10-24% participated in planning 
25-49% participated in planning 
50-99%) participated in planning 
100% participated in planning 



12. During the past three years, what PERCENTAGE of your ENTIRE teaching 
staff (including teachers in areas such as krt, reading, music, mathematics, 
science, physical education, etc.) have received inservice TRAINING in 
CITIZENSHIP/SOCIAL STUDIES from your school or district? 



ERLC 



No citizenship/social studies inservice training offered 

None received training 

1-9% received training 

10-24% received training 

25-49%) received training 

50-99%, received training 

100% received training 
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13. For each of the foUovving areas, indicate whether or not the students in 
your school are given instruction in that area or are provided with 
activities that would help them develop knowledge and skills in that area. 

A. ACQUIRING INFORMATION 







A part of 
instruction 
or activity 


NOT a part 
of instruction 
or activity 


1. 


Use of the senses to 
obtain information 




CTD 


2. 


Use ^ a variety of sources 
to obtain information 


c — \ 


c — > 


3. 


Use of a variety of 
techniques (e.g., 
interviews, polls, surveys) 


crz) 




B. 


USING INFORMATION 


y 

A part of 
instruction 
or activity 


NOT a part 
of instruction 
or activity 


1. 


Organizing information 
(e.g., arranging, grouping, 
relating, analyzing) 


CD 




2. 


Applying information (e.g.. 
inferring, predicting, 
testing alternatives) 


CZ) 


CD 


3. 


Decision making and 
problem solving 


CD 


CD 


4. 


Critically evaluating 
information 


CD 


CD 
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13. (Continued) 



C. UNDERSTANDING INDIVIDUAL DEVELOPMENT 







A part of 
instruction 
or activity 


NOT a part 
of instruction 
or activity 


1. 


Examining the 
individual's beliefs, 
values and behaviors 


CZZ) 


CZZ) 


2. 


Individual developnnent 
(setting goals, determining 
consequences of goals) 


CZZ) 


CZZ) 


3. 


Graphic and oral 
communication 


CZZ) 


CZZ) 


4. 


Interpersonal communication 


CZZ) 


CZZ) 


5. 


Interaction in groups 


CZZ) 


CZZ) 


6. 


Relating to people of 
different cultural 
perspectives 


CZZ) 


CZZ) 


D. 


UNDERSTANDING THE WAYS HUMAN BEINGS ORGANIZE 

t. 






A part of 
instruction 
or activity 


NOT a part 
of instruction 
or activity 


1. 


Ways in which people are 
interrelated (e.g., 
economically, environmentally, 
historically, politically, 
culturally, etc.) 




CZZ) 


2. 


Organizations of human 
societies (e.g., nature of 
institutions and groups) 


CZZ) 


CZZ) 
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13. (Continued) 



D. 



UNDERSTANDING THE WAYS HUMAN BEINGS ORGANIZE 
(Continued) 



A part of"-^ 
instruction 
or activity 



NOT a part 
of instruction 
or activity 



Relationships between 
individuals and groups 
(e.g., roles in groups, 
effect of groups on 
individuals) 



Relationships betv^^een 
groups (e.g., cooperation 
and conflict among groups) 



Global concerns (e.g., 
food, population, disease, 
racism, conflict, human 
rights) 



6. Human rights v^^orldv^^ide 

(e.g., acceptance and y 
appreciation of universal cz) ' CZ) 

differences, equitable 
treatment, consideration 
for interest of others) 



E. UNDERSTANDING THE DEVELOPMENT OF THE UNITED 
STATES 



A part of ' NOT a part 

instruction of instruction 

or activity or activity 



1. Principles and purposes 






of the U.S. government 




CZ) 


2. Organization and operations 






of U;S. government (federal. 


CZ) 


CZI/ 


state, local) 
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E. UNDERSTANDING THE DEVELOPMENT OF THE UNITED 
STATES (Continued) 



A part of 
instruction 
or activity 



NOT a part 
of instruction 
or activity 



3. 


Political decision making 
in the U.S. 




CID 


4. 


Electoral processes in 
the U.S. 


(ZD 


CID 


5. 


Legal system in the U.S. 


C=D 


CID 


6. 


Rights of individuals in 
the U.S. 


CID 


CID 


7. 


Civil and criminal justice 
systems in the U.S. 




CID 


8. 


Support of justice, and 
rights for all individuals 


CID 




9. 


Economics in the U.S. 


CID 


CID 



14. FOR HIGH SCHOOLS ONLY: 

How many carnegie units are students required to have in citizenship/ 
social studies curriculum for graduation in your school district. 

CZD More than one carnegie unit 

CD One carnegie unit 

CZD One-half carnegie unit 

CZD Less than one-half carn'egie unit 

CZD None required 



THANK YOU FOR YOUR COOPERATION. 



School Name: 







Age 
Class 
2 


Age 
Class 
3 


P.S.f. 
No. 




No. 




School 
No. 






















CID CO 
CIDCD 
CIDCSD 
CDCZD 
CaDOD 
CDCD 
CZDCID 
CDCED 
CDCED 




r""o ) 








dJCUCLJ 




dDCIDCID 
CIDdDdD 
CD OD CO 
CDCDCD 
CJDCJDdD 
dDCIDCID 
dDdDdD 
CDCDCD 
CDCDCD 








f J J 






99899968 
89999989 
89999899 
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ADJUSTMENT CF RESPONDENT WEIGHTS BY SMOOTHING TO 
REDUCE RANDOM VARIABILITY OF ESTIMATED 
POPULATION PROPORTIONS 



Background 

As noted elsewhere, a weight is assigned to every indi- 
vidual who responds to an exercise administered in - an as- 
sessment. The weight is the reciprocal of the probability 
of selection of the individual with adjustment for nonres- 
ponse. It estimates the nunber of similar people that that 
individual represents in the age population. The sun of the 
weights of all individuals at an age level responding to an 
exercise is an estimate of the total nunber of people in 
that age population in the year that the exercise was as- 
sessed. Similarly, the sun of weights for all individuals 
who took the exercise and who also are members of some demo- 
graphic category (such as blacks) gives an estimate of the 
nunber of people in the age population, for the year, v^o 
are also members of the category. The ratio of the two to- 
tals estimates the proportional representation of the demo- 
graphic category in the age population for the given year. 

For each of the assessed age populations in each assess- 
ment year, separate estimates of the proportional represen- 
tation of the various demographic subgroups are provided by 
each booklet administered to that age group in that year. 
Due to random sampling variability, the estimates of popula- 
tion proportions for a given year based on single booklets 
administered in the year will vary. In addition to whatever 
trends in population proportions over time that might exist, 
there is also random sampling variation in these proportions 
from year to year . 

It is desirable to reduce the random variability of popu- 
lation proportions as much as possible, since this variabil- 
ity has an effect on performance estimates. For example, 
the percentage of acceptable responses for an age group is a 
function of the relative proportions of high-performing and 
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low-performing groups. If the relative proportions of these 
groups are very different in different assessments due to 
sampling variability, then a portion of the change in per- 
centage of acceptable responses for an age group could be 
attributable to yearly sampling difference in the relative 
proportions of high- and low-achieving groups. 

In addition to reporting performance estimates for an age 
group as a whole. National Assessment also reports perform- 
ance for various sub populations , such as vAiites or blacks. 
Because variability of s.ubgroups within these subpopulations 
(such as white males and females within the vhite subpopula- 
tion) influence the performance estimates for the subpopula- 
tions, it is desirable that fluctuations of proportions of 
all subgroups of each subpopulation be reduced as much as 
possible. 

For each age and year, each of the various booklets ad- 
ministered will provide estimates of a given population pro- 
portion. Since these estimates are subject to booklet to 
booklet variability, a better estimate of the population 
proportion, which will have reduced variability, is obtained 
by combining the information ft-om all booklets. ftowever, 
these proportions vary from year to year due to random sam- 
pling variability or systematic differences *in sampling pro- 
cedures. An even better estimate of population proportions 
for any single year can be obtained by smoothing the propor- 
tions over several assessment years. Ihe word "smoothing" 
is used here in the sense of fitting a smooth curve to a se- 
quence of nunbers by robust/resistant procedures. Snoothing 
estimates of population proportions . reduces a large portion 
of the sz jpling variability while preserving, as far as pos- 
sible, actual trends occurring in the age population. 



After the population proportions have been smoothed, ad- 
justed weights are derived for the assessed individuals so 
that the population proportions computed using the adjusted 
weights are equal to the smoothed proportions. The adjusted 
weights are then used for all analyses, such as estimation 
of performance. 

The result of the smoothing and weight-adjusting process 
is that both adjusted performance estimates and changes in 
those estimates appear to be somewhat less susceptible to 
sampling variability, both across and within years. 
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Snoothlng ' Prooedures Used by National Assessment 

The most direct way to amooth proportions is first to 
classify people into mutually exclusive mul'ti-way cells on 
the basis of their membership in categories of various im- 
portant variables and then to amooth the proportions within 
each of. the resulting multi-way cells across years. Unfor- 
tunately, this procedure tends to produce a large number of 
cells with few people and consequently quite unstable esti- 
mates of smoothed proportions. 

To circunvent this difficulty. National Assessment has 
utilized various, smoothing procedures since the 1976-77 as- 
sessment. Each of these procedures, which are all basically 
weighting-class adjustments applied independently to each 
age, is designed to control, to varying degrees, fluctua- 
tions in certain key subgroups while avoiding, as much as 
possible, instabilities due to small cells. 

The procedure used for the 1976-77 assessment was a 
weighting class adjustment 'applied Independently to each age 
and ^ reporting variable (nation, region, sex, etc.). The de- 
tails of the procedure are given in Appendix B of Technical 
Report 08-S-21: Three Assessments of Science, 1969-77: 
Technical Summary. While this procedure performs well, it 
is complicated and requires large amounts of time and com- 
puter resources to implement. By independently -smoothing 
proportions within each reporting variable, it was possible 
to produce good estimates of the marginal proportions of 
people within each category of the variable, while disturb- 
ing as little as possible the relationships between other 
reporting variables within the adjusted variable. However, 
this meant that each individual had a different adjusted 
weight for each reporting variable under consideration. 
While this presents no problem for the estimation of per- 
formance within a reporting variable, the multiplicity of 
weights definitely complicates analyses, such as regression, 
that involve several variables. 
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Because of the complexity of the procedure used in 

1976- 77, a different and simpler procedure was adopted in 

1977- 78* This procedure is detailed in Appendix F of Report 
09-MA-40, Procedural Handbook ; 1977-78 Mathematic 3 
Assessment , The 1977-73 smoothing procedure produced a sin- 
gle adjusted weight for each individual, and hence greatly 
reduced the complexity of subsequent analyses of performance 
data. The 1977-78 procedure involved applying a weighting 
class adjustment independently to each age. The weighting 
classes ^^jy*ich were different at each age, consisted of in- 
dividuals who were alike on certain demographic characteris- 
tics and who would be expected to have similar educational 
achievement characteristics. There were around seventy ad- 
justment cells used for each age. 

a 

Although the 1977-78 procedure prod^uced acceptable re- 
sults. National Assessment in 1978-79 adopted yet another 
procedure which we believe has the best characteristics of 
the three procedures used. The new (1978-79) procedure, 
which is detailed below, has several advantages. 

1. It produces a single adjusted weight for each indi- 
vidual. 



2. It affords good control on the distribution of pro- 
portions of certain key variables. 

3. It tends to produce stabl'e performance estimates. 
i|. It is the easiest to implement. 

Even though adjusted weights using this, ^procedure differ 
slightly from the corresponding adjusted weights from the 
other procedures, we intend to use weights obtained using 
this procedure for til future analyses of data assessed in 
earlier years. This is simply because we believe these 
weights to be the best available. 
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• The current anoothing Procedure 

The first step in the 1978-79 anoothing procedure 
involved the partitioning of the population of age' class 
eligibles into the six smoothing cells given in Exhibit 2,1 
The same cells were used for all ages. 



EXHIBIT 2. 1 



anoothing Cells Used for the 1978-79 Procedure 

■ I. ' .^^'^'^ 

Community Size (OS) 



Cell 


Race 


Reg ion 


1 


White 


All 


2 


White 


All 


' 3 


White 


All 


4 


Black 


SE 


5 


Black 


NOT SE 


6 


Other 


All 



Big City + Fringe 

Mediun City (MC) 

anall Flaces (SP) 

All 

All 

All 



(BC + FR) 



8i 



/ 
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Then, for each age and every year, the proportion of the 
population in each of the cells was estimated. For a given 
age and year, the proportion of the population in a particu- 
lar cell was computed as the sum of weights of , all respon- 
dents assessed in the given year who were of £h^specified 
age and who belonged ^in the cell, divided by the total of 
the weight of all respondents of the given age assessed in 
that year. Proportions for 9-» , 13- and 17-year-olds are 
shown in Exhibits 2.2A, B and C, respectively. 

Each of the six cells was composed of a sequence of esti- 
mated population proportions corresponding to the various 
years of assessment. Each such sequence of proportions was 
then smoothed by fitting robust/resistant lines. Using data 
from the Census and Current Population Surveys, trends in 
enrollment by age and race and by age and region were ob- 
tained. The data from these surveys were adjusted to corre- 
spond with NAEP definitions as much as possible. The resis- 
tant lines within the smoothing cells were constrained to 
satisfy the trends from the Census and CPS data. 

The final step in the smoothing procedure was to adjust 
the respondents' weights to be consistent with the smoothed 
proportions. Since each respondent takes only one booklet, 
the weight adjustments were done independently for each 
booklet. For a given age, year and booklet, population pro- 
portions using the original weights were obtained for each 
of the smoothing cells. Then the weights of all respondents 
of a given cell were multiplied by the ratio of the smoothed 
cell proportion to the proportion using the original 
weights. This produced the adjusted weights which are used 
in all analyses. 




EXHIBI'' k:^ 

Un^obthed Frorortions fay & nothing ' -r.il and Yeer 







for 


Nine-Ye.i 








CELL 


1 




3 


i| 


*.'•' 


6 


RACE 


WHITE 


WHITE 


WHITE 


BUCK 




OTHEF: 


REGION 


AL^ 


ALL 


ALL 


SE 


liOT SE 


ALL 


CS 


EC+f R 


MC 


SP 


' :.i 


AIL 


A 


YEAR 














69-70 


• 335^5 


.11333 


.36797 


- '■ 3937 


.0661 VI 




70-7 1 « 


. 32532 


. 12165 


.351514 


.0 jHono 


.077328 


.06618^ 


71-72 


.31743 


. 10852 


.38^9 


.. C6J266 


.0714271 


.058037 


72-73 


. 52555 


. 10957 


. 35? !S 


.062JH3 


.08186'< 


.06704^ 


73-7'^ 


. 3 152i) 


. 13755 




.056249 


.089667 


.057184 






. 12'<22 


. 39229 


< u 0 < I 0 c 






75-76 


.27167 


. ^^H3^ 




.050^^0 


.059552 


.089095 


76-77 


.27819 


. 11 i.:6 


.^2390 


.057192 


.069709 


.059827. 


77-78 


. 33610 


.06958 


. 38770 


.070198 


.066873 


.069632 


78-79 


.28557 


.08517 


.390 '40 


.033710 


•0772 5 9 


.077959 


79-80 


.23366 


.11382 


.^45130 


.0^42650 


.073036 


.0856(0 


81-82 


. i)1792 


.05381 


.29220 


.07951 6 


.078769 


.077786 


NOTES: 














« In 


1970-71, 


Hispanlcs were 


Includ ed 


in the 


"White" 



category; they are in "Other" for all other assessments. 



ERIC 
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EXHIBIT 2.23 



liismoothad FV-oportions by anoothing Cell and Year 
for Thirteen-Year-Olds 



CELL 
RACE 
REGION 
CS 


1 

WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 
MC 


3 

WHITE 
ALL 
SP 


4 

BLACK 
SE 
ALL 


5 

BLACK 
NOT SE 
ALL 


6 

OTHER 
ALL 
ALL 


YEAR 














69-70 


.31308 


. 13724 


.35597 


.059331 


. 073373 


. 061090 


70-71 • 


.31105 


. 10757 


. 37370 


. 058330 


.088536 


.060869 


71-72 


.35923 


. 11459 


. 36682 


. 046199 


.0560S1 


.057165 


72-73 


•33276 


- 11460 


.36012 


. 052248 


.074108 


.066245 


73-7'* 


. 32757 


. 12686 


.36269 


. 050682 


.075044 


. 057221 


74-75 


. 29797 


. 11200 


. 40792 


.OMOI9 


. 068247 


.052903 


75-76 


.24279 


. 10656 


.47169 


.047466 - 


.046586 


.084974 


76-77 


.27905 


. 09100 


.42018 


. 050036 


. 108012 


.051784 


77-78 


.36593 


. oegi'* 


.36030 


.065470 


. 065295 


. 073942 


78-79 


.33351 


0 08365 


.33756 


. C75508 


.087461 


08239V 


79-80 


.22430 


. 16140 


.40484 


. 040344 


.090701 


.078483 


81-82 


.40147 


.07650 


.29938 


-059464 


.082718 


.080481 


NOTES: 
» In 


1970-71, 


Hispanic 3 were 


included 


in the 


"White' 



-category; they are in "Other" for all other assessments. 
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EXHIBIT 2.2C 



Unanoothed a-oportions by anoothing Cell and Year 
for Seventeen-Year-Olds 



CEI^L 
RACE 
REGION 
CS 


1 

WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 
MC 


3 

WHITE 
ALL 
SP 


1 

BLACK 
SE 
ALL 


5 

BUCK 
NOT SE 
ALL 


6 

OTHER - 
ALL 
ALL 


YEAR 














69-70 


•33826 


• 157'<3 


•37913 


..036070 


•061890 


• 027287 


70-71* 


•37123 


. 12382 




•015810 


•067916 


•019850 


71-72 


•35383 


• 11700 


•37216 


. 017010 


•059155 


. 050627 


72-73 


•33776 


• 11225 


.375C5 


•018503 


•071175 


.055032 


73 -T* 


•35497 


• 1'<873 




.013113 


•066038 


. 0102f4i: 


7'»-75 


•3^387 


• 11761 


•37817 


•057713 


•063378 


.039311 


75-76 


•25909 


• 118'<2 


•16370 


• 013622 


•052995 


.062216 


76-77 


•31808 


• 11'J27 


•10588 


.013133 


•072569 


• 0161 15 


77-78 


• 10073 


. 06;^ 2 6 


•36979 


• 050921 


•060797 


. 055582 


78-79 


• 3627'* 


. 09967 


•35^07 


•052836 


•059185 


.071276 


79-80 


• 2'<808 


• 17037 . 


.10955 


•039871 


.072115 


. 060060 


81-82 


• 38273 


• 0'<901 


•30715 


.061539 


•O993I6 


. 096929 


NOTES: 
» In 


1970-71. 


Hispanic 3 were 


included 


in the 


"White" 



category; they are in "Other" for all other assessments. 
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Adjustment of Weights by Users 

Snoothed population proportions for 1969-70 through 
1979-80 are given in Exhibits 2.3A, B and C for 9-, 13- and 
17-year-.olds, respectively. These are the proportions used 
in analyses involving 1979-80 reading/literature data and 
other NAEP analyses conducted betv?een late 1980 and late 
1982. For exa^iple, the smoothed population proportion of 
9-year-old3 in smoothing cell tw (whites in mediun cities) 
for I972-.73 is .11518. Note that the 1970^71 entries are 
blank. In that assessment, iriispanics were included in the 
"white" classification rather than "other" or a separate 
category. Consequently » s-aoothed proportions have not been 
used by National Assessment for analyses of the 1970-71 
data. 

To iidjust respondent knights to be consistent vdth the 
smoothed proportions, the following procedure is followed: 

I0 For each booklet, classify the respoj^eP^nts according 
to smoothing cell and obtain the^r^ population pro- 
portioms for -each cell. For ex*§raple, the raw propor- 
tion for 3 booklet of 9-year-clds in sr^oothing cell 
four is thj total of the weights of all 9-year-olds 
in the l>ook].et who are Bleck and in the Southeast re- 
gion, divided by the total of the weights of all re- 
sponderts to the booklet. 

2. For each booklet and smoothing cell, obtain a weight 
adjustment factor as the ratio of the smoothed popu- 
lation proportion (for the appropriate age, year and 
smoothing cell) over the raw population proportion. 

3. Ihe adjusted weight for an individual is the product 
of that Individual's original weight and the appro- 
priate adjustment factor* 




So 
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EXHIBIT 2.3A 



anoothed Proportions by Snoothing Cell and Year 





for Nine-Year- 


-Olds from 


1969-70 to 


1979-80 




CELL 
RACE 
REGION 
CS 


1 

WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 
MC 


3 

WHITE 
ALL 
SP 


i| 

BLACK 
SE 
ALL 


5 

BLACK 
NOT SE 
ALL 


6 

OTHER 
ALL 
ALL 


YEAR 














69-70 


.32930 


. 12576 


. 35^56 


.0535*4^ 


.07^^*86 


.06235 


70-71 e 














71-72 


•32322 


. 1 1 772 


. 36^*72 


.056238 


.07^4252 


.06385 


72-73 


.31650 


. 11518 


. 37200 


-056790 


.07^930 


.06*460 


73-7'< 


. 3 09''' 7 


. 11265 


. 37928 


. 0573^2 


.075608 


.06535 




.30305 


. 1101 1 


. 38656 


.05789^ 


.076286 


.0^:^610 


75-76 


.29632 


.rC757 


. 39385 


.058'4'47 


.076963 


.06685 


76-77 


.28S6G 


. 10503 


. ICII3 


.058999 


.0776'I1 


.06760 


11 -Id 


.28287 


. 102*19 


. 108^1 


.059551 


C.r!319 


.06835 


78-79 


.27615 


.09996 


.^569 


.060103 


. 078997 


.06910 


79-80 


.269'<3 


.097'<2 


. 12298 


.060656 


. 07967*4 


.06985 


NOTES: 
« 197C 


1-71 data 


omitted because 


of non- 


standard 


racial 



ethnic definitions- 



PAGE 2- 12 



APPENDIX 2 

EXHIBIT 2. 3B 



anoothed Proportions by Snoothing Cell and Year 
for Thirteen-Year-Olds from 1969-70 to 1979-80 



CELL 

Pace 

REGION 
CS 


1 

WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 
MC 


3 

WHITE 
ALL 
SP 


4 

BLACK 
SE 
ALL 


5 

BLACK 
NOT SE 
ALL 


6 

OTHER 
ALL 
ALL 


YEAR 














69-70 


. 3'c002 


. I3O88 


.37030 


.051270 . 


.067330 


.06020 


70-71 » 














71-72 


. 32792 


.11055 


. 37793 




.0*. ^z<}Z 


.C "180 


72-73 


.32317 


. 10982 


.38101 






. 06260 


73-7^ 


•31841 




.38409 




.072418 


.06540 


7'^-75 


.31366 




.36717 




.073931 


. 06420 


75-76 


.30890 




=.39026 


.052756 


.075444 


.06500 


76-77 


.30415 


. 10691 


• 3933'* 


.052344 


.076956 


.06580 


77-78 


.29940 


. 10618 


.39642 


. 05293 1 


.078469 


.06660 


78-79 


.29464 


. 10546 


. 3995u 


.053018 


. 079982 


.06740 


79-80 


.28989 


. 10473 


. "D;" 


.053105 


.081495 


.06820 



NOTES: 

^ 1970-71 data omitted because of non-standarc racial/ 
ethnic definitions. 
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EJvnlBIT 2.3c 

anoothed Proportions by anoothing Cell and Year 
for Seventeen-Year-Olds from 1969-70 to 1979-80 



PITT T 

RACE 

REGION 

CS 


1 

WHITE 

ATT 

ALL 
BC+FR 


WHITE 

ATT 

ALL 
MC 


3 

WHITE 

ATT 

ALL 

SP 


l| 

BLACK 

St 

ALL 


5 

BLACK 
NOT SE 
ALL 


0 

OTHER 

A 7 T 

ALL 
ALL 


YEAR 








• 






69-70 


.3'»050 




. 36862 


.041H8 


.0581 10 


.0^658 


70-71 • 














71-72 


.35766 


. 1199'* 


.37040 


.044424 


.059726 


.0^1782 


72-73 


.35192 


. 11939 


•37379 


.045060 


.061370 


.0^8^^ 


73-74 


.3'<618 


. 1 ;88'< 


.377i:' 


-045697 


.063013 


.04906 


7H-75 


.3'ioyy 


.11829 


.3805' 


.0^6333 


. J64657 


.04968 


75-76 


.33'<70 


. 1177'* 


.38396 


.046969 


.006301 


.05030 


76-77 


.32896 


, 11718 


.38736 


.047606 


.06794 4 


. 05092 


77-7? 


.32322 


. 11663 


.39075 


.048242 


.'y:9'=88 


.05154 


78-79 


.317'<8 


.11608 


.39414 


.048878 




052 16 


79-80 


.31174 


.11553 


.39753 


.049514 


.072876 


.05^78 


NOTES: 















* 1970-71 data omitted because of non-standard racial/ 
ethnic definitions. 
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Changes In Snoothed Proportions as New Assessments New 
Assesanents are Completed 



Every time an assessment is complttKl, a new time point 
i3 added to each of the sequences of population proportions 
within the smoothing cells. This ineans that, even though 
robust/resistant procedures are used, the addition of a new 
point may somewhat change th€^ values of smoothed proportions 
for prior years. Additionally, any changes in methodology 
will impact the escimates. 

Ihis means that the smoothed proportions, with the addi- 
tion of the next assessment data, are apt to differ somewhat 
from the corresponding smoothed proportions without the new 
data. National Assessment has adopted the philosophy that 
the smoothed proportions, based on all currently available 
data using the best available algorithm, are the best avail- 
able data. Therefore, all subsequent analyses, for any 
year, will be done using this best available information, 
even though this may produce estimates which slightly differ 
from prior values (before implementation of the newest 
smoothed proportions) . 



In late 1982, newly smoothed proportions were estimated 
for the p^?riod 1972-73 to 1981-82. These proportions are 
shOHri in Exhibits 2. ^iA, B and C for three ages. They were 
used for all analyses involving the 1981-82 assessment of 
citizenship/ social studies, mathematics and science. It 
should be noted that seme of the trend lines for cells in 
Exhibits 2. 3A, B and C re quite different from those for 
cells in Exhibits 2.Hk, B and C. Additional work may be re- 
quired to obtain sati sfsc torily smoothed proportions for the 
entire time period, 1969-70 to 1981-82. 
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EXHIBIT 2.'JA 



Snoothed Proportions by anoothint, ' and Year 
for Nine-Year-Olds from 1972-' oo 981-82 



CELL 
RACE 
REGION 
CS 


1 

WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 
MC 


3 

WHITE 
ALL 
SP 


^ 

BLACK 
SE 
ALL 


5 

BLACK 
NOT SE 
ALL 


\ 6 

OTHER 
ALL 
ALL 


YEAR 














72-73 


.29556 


. 1322« 


. 37592 


.05197 


.07676 


.06165 


73-7 U 


.29807 


. 12530 


.37837 


.05570 


.07726 


.06510 


71-75 


. 3CO?9 


.11836 


.38081 


.05612 


. 07777 


.06615 


75-76 


.30311 


. ^1^^2 


.38325 


.05711 


.07828 


.06690 


76-77 


.30563 


. ^o^^7 


. 38570 


. 05787 


.07878 


.06765 


77-78 


.30815 


.09753 


.38811 


.05859 


.07929 


.06810 


78-79 


.31066 


.09059 


.39059 


.05932 


.07979 


.06915 


79-80 


■313I8 


.08365 


.39303 


.06001 


.08030 


.06990 


81-82 


.31822 


. 06977 


. 39792 


.06119 


.08131 


.07110 



ERIC 



Si 



J 
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EXHIBIT 2.'JB 



aioojthed ft-oportions by anoothing Cell and Year 
for Ihir teen-Year-Olds from 1972-73 to' 1981-82 



CELL 


1 


2 


3 


1* 


5 


6 


RACE 


WHITE 


WHITE 


WHITE 


BLACK 


BLACK 


OTHER 


«?EGION 


ALL 


ALL 


ALL 


SE 


NOT SE 


ALL 


CS 


B C-»F 


MC 


SP 


ALL 


ALL 


ALL 


YEAR 














72-73 


. 29977 


. 11231* 


.1*0190 


. 05530 


.06810 


.06260 


73-7H 


. 30298 


. 11076 


. 39786 


. 05H80 


.07020 


. 063 1'O 


7U-75 


. 30620 


. 10918 


. 39382 


.05H31 


. 07229 


.061*20 


75-76 


. 309'<2 


. 10760 


. 38978 


. 05382 


.071*38 


.06500 


76-77 


.31263 


. 10602 


. 3857i< 


.05332 


<;■-" 

.0761*8 


.06580 


77-78 


.315o5 


. 101*il5 


.38170 


.05283 


.07857 


.06660 


78-79 


.31907 


. 10287 


-37766 


.05231* 


.08066 


.0671*0 


79-80 


. 32228 


. 10129 


.37362 


.05181* 


. 08276 


.06820 


81-82 


. 32872 


.09811* 


.36555 


. 05086 


.08691* 


.06980 



ERIC 
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EXHIBIT 2. HC 



anoothed Proportions by SnosJ thing Cell and Year 
for Seventeen-Year-01d3 from 1972-73 to 1981-82 



CELL 

nACll 

REGION 

CS 


. 1 
WHITE 

ALL 
BC+FR 


2 

WHITE 
ALL 

MC 


3 

tiff T^p 

WHITE 
ALL 
SP 


1 

BLACK 
SE 
ALL 


5 

BLACK 
NOT SE 
ALL 


. 6 
OTHER 
ALL 
ALL 


lEAR 














72-73 


• 329'*'* 


• 11852 


.39711 


.017036 


.059311 


.01812 


73-7'* 


•33169 


.11815 


.39235 


.017118 


.061512 


.01901 


7'»-75 


•33395 


11778 


.38757 


.017261 


.063679 


.01966 


75-76 


•33620 


. 117'*2 


.38278 


.017373 


.065817 


.05028 


■76-77 


•33816 


• 11705 


.37799 


.0"7185 


.068015 


. 05090 


77-T8 


. 3^*072 


.11668 


.37320 


.017598 


.070182 


.05152 


78-79 


•3^297 


• 11631 


.36812 


.01771'" 


"50 


.05211 


79-80 


•3^523 


.11591 


.36363 


.017823 


.071517 


.05276 


81-82 


.3'*'<79 


.11358 


.31901 


.017367 


.077735 


.06710 
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GLOSSARY OF NATIONAL ASSESSMENT TERMINOLOGY 

ACCEPTABLE RESPONSE - Any response to an exercise that denj- 
onstrates achievement of the objective measured by that 
exer<2ise. 

ADMINISTRATION MODE - The way in which an exercise is admin- 
istered . ' Exercises are administered in either group or 
individual mode. See "INDIVIDUAL AiSnlNISTRATION" and 
"GROUP AEMINISJRATION." 

ADMINISTRATION TIMETABLE - Tfie time periods dui-ing a ^chool 

ye ar^ when the various age groups are assessed. The time 
periods, are: 

October-December 1 S^y^ar-olds 

January-March 9-.year-old3 

March-May 17-year-olds (in-school) 

Spring-Summer 17*year-olds (out-of-school) 

The assessment time period for young adults, ages 26-35 has 
varied from year to year. 

ACHIEVEMENT CLASS VARWBLE (ACV) - A classification of a re- 
spondent into performance quartiles based on his or her 
performance' on achievement items in^a particular booklet. 
The definition of an achievement item is developed for 
'-each assessment year and may differ slightly from year to 
year. However, in general an achievement item is any cog- 
nitive item for which there f.3 a response that is keyed as 
correct . 

AFFECTIVE EXERCISES ► Attitude, experience and other items 
which lack an ''acceptable" response or are considered to 
be important a.^pectit of a learning area, but not appropri- 
ate for inclusion in knowledge or skill sunmaries. 

AGE CLASS, AGE GROUr OR /.'jE LEVEL - One of the age levels 
sampled in the dooeS3jne^>t « Each group is assigned a nu- 
meric value and an alphabetic designation, as follows: 

Age class 1 = N = 9-year-olds 

Age class 2 = T = 13-year-olds 

Age clas3 3 = S = IT-year-olds 

Age class H = A = Young Adults (ages 26-35) 




2 
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AGE-ELIGIBLE - Any person meeting the age definition for an 
assessment. Birthdate ranges for each age group in the 
1969-70 to 1979-80 assessments were: 







Aae 1'=? 


Afle 17 


Adult 


1969-70 


I960 


1956 


10/51-9/52* 


7/33"6/i<3 


1970-71 


1961 


1957 


10/5 3-9/5 1»» 


1J/35-3/1J5 


1971-72 


1962 


1958 


o10/5i»-9/55» 


4/36-3/1J6 


1972-73 


1963 


1959 


10/5 5-9/5 6» 


1937_19H6 


1973-74 


1961J 


I960 


10/56-9/57 


1938-1 9'I7 


1971J-75 


1965 


1961 


10/57-9/58 




1975-76 


1966 


1962 


10/58-9/59 




1976-77 


1967 


1963 


10/59-9/60 


1941-1950 


1977-78 


1968 


1961J 


10/60-9/61 


«* 


1978-79 


1969 


1965 


10/61-9/62 


«« 


1979-80 


1970 


1966 


10/62-9/63 


«« 



totes; 

» In the first ^ assessments, out-of-school IT-year-olds 
could be up to 12 months older than shot/n in the table 
if they were not enrolled in school during March of the 
year prior to the assessment. 

Adult- assessments were not conducted in these years. 

iiSSF.SGMENT - The documentation of the progress in knowledge, 
skills and attitudes of Anerican youth. Measures are 
taken at periodic intervals for each learning area, with 
the goal of determining trends and reporting the findings 

- to the public and to the education community. 

ASSESSMENT YEA'^? • Annual assessments are nunbered 
sequentially, starting with the 1969-70 assessment . 
Since the 1980-81 school year, it has been necessary to 
change to a biennial data collection due to budget 
constraints. BACKGROUND QUESTIONS - Several types of 
background questions are included in exercise booklets. A 
common set of questions about educational materials in the 
home, level of parental education, etc., have been include 
ed on the last page of the exercise booklet in every as- 
sessment. Beginning in 1975-76, 17-year-olds have been 
asked additional background and demographic questions, 
some of whioh are derived from the National Longitudinal 
Study of the High School Qass of 1972. Finally, learning 
area-specific background questions are sometimes included. 
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BACKCROUND QUESTIONNAIRE • This term is used for the form 
U5e<j to collect background information from out-of-school 
17-year-old3 and young adults. 

BACKGROUND VARIABLE - See "VA^.iABLE." 

BOOKLET - Items (exercises) are presented to responoents in 
booklets. Booklets are designed to be scored by optic^il 
scanning machines. Each booklet contains instructions on 
answering items, practice items, th<^ assessment items and 
background questions. Each booklet contains approximately 
30-35 minutes of assessnent items and 10-15 minutes of in- 
troductory material and background questions. A booklet 
typically includes exercises of varying difficulty from 
different objectives of the learning area(3) being as- 
sessed. If more than one learning area is being assessed, 
the booklet may contain exercises from more than one 
learning area. The terms "booklet" and "package" are sy- 
nonomous and may be modified by any of the following with- 
out change in meaning: assessment, exercise « Item or re- 
spondent. 

BOOKLET WEIGHT - See "WEIfflT." 

CATEC»RY (SCORING) - A classification of a response to an 
\ open-ended exercise. See "SCORING GUIDE." 

CATEGORY WITHIN A VARIABLE - A subclassiflcation within a 
variable. For example, male and female are categories of 
the^variable sex. j 

CENSUS DIVISION - For most assessments , respondents are 
classified according to U.S. census divisions, based on 
location of their school or home (for household surveys of 
adults and out-of-school 1 T-year-olds) . The categories 



are : 



1 = New England 



5 = South Atlantic 



Connecticut 
Maine 

Massachusetts 
New Hampshire 
Rhode Island 
Vermont 



Delaware. 

District of Colunbla 

Florida 

Georgia 

Maryland 

North Carolina 

South Carolina 

Virginia 
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2 r Middle Atlantic 
New Jersey 
New York 
Pennsylvania 



East North Central 
Illinois 
Indiana 
Michigan 
Ohio 

Wisconsin 
West North Central 



Iowa 

Kansas 

Minnesota 

Missouri 

Nebraska 

North Dakota 

South Dakota 



APPENDIX 3 
West Virginia 
East South Central 



Alabama 
Kentucky 
Mississippi 
Tennessee 

West South Central 



Arkansas 
Loui siana 
Oklahoma 
Texas 



8 = Mountain 



Arizona 

Colorado 

Idaho 

htontana 

Nevada 

New Mexico 

Utah 

Wyoming 
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9 = Paoifio 
Alaska 
California 
Hawaii 
Oregon 
Washington 

COGNITIVE EXERCISE - An item measuring behaviors in the cog- 
nitive domain of Bloom's Taxonomy. 

COMMUNITY SIZE - Also called derived size of community 
(DOC). Schools (and households in the adult assessments) 
are classified in four categories using Census, ZIP Code, 
and atlas information: 

1 s Big City: I97O population of 200,000 or more 

2 = Big City R"inge: Schools/ segments around big cities 

(in the 1970 Census Urbanized Area of a Big City) 

3 s Median City: 1970 population between 25f000 and 

200,000 and not in 2 above 
,M = Small Places: All other places 

Out-of-school 17-year-olds sampled from school lists of 
dropouts and early graduates receive the same community 
size classification as the school received in the in- 
school assessment. Note that this variable is not identi- 
cal to either thfe size of community sample stratification 
variable or the principalis questionnaire size of communi- 
ty served variable. 

CONTENT AREA - See "LEARNING AREA." 

DELTA P-VALUE - NAEP terminology for the difference in per- 
centages between a reporting group (such os males) and the 
whole nation on a particular response option or category 
(usually the acceptable response). 

DERIVED VARIABLE, DERIVED EXERCISE PART, DERIVED VALUE - 
Data derived by combining responses to two or more vari- 
ables or by combining values of one variable. 
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DIFFICULTY LEVEL - The percentage of acceptable responses to 
an exercise • 

DS OR DISTRICT SUPERVISOR - An individual employed to manage 
the In-school data collection. Each DS works In a specif- 
ic section of the country contacting schools and making 
arrangements for the assessment * The DS also administers 
booklets to respondents* 

DOC OR DERIVED SIZE OF COMMUNITY - See "COMMUNITY SIZE." 

EA OR EXERCISE ADMINISTRATOR - An individual hired locally 
and trained by the district supervisor to administer book- 
lets to in-school respondents. 

ELIGIBLES - Individuals who meet the criteria for inclusion 
in the assessment. See " INELIGIBLES." 

EXERCISE - A task designed to measure an objective. In some 
- cases the task is expressed by a single question ♦ In other 
cases the task is expressed by several related questions. 
Some tasks are direct measures of performance, e.g., play- 
ing a musical instrunent or writing a letter. Others are 
indirect-measures, e.g., answering a multiple choice ques- 
tion or writing a short answer to a question. Because- 
NAEP does not administer "tests" but surveys educational 
achievement over time, the term exercise is often used in- 
stead of the terms item or test item. Item and exercise 
are used synonomously in the NAEP docunentation. 

EXERCISE BOOKLET - See "BOOKLET." 

EXERCISE PART - See "ITEM PART." 

EXERCISE VARIABLE - See "VARIABLE." 

EXERCISE POOL - The entire set of exercises prepared for a 
learning area. This set includes the exercises from pre- 
vious assessments used for measuring change, exercises de- 
veloped for previous assessments but not used due to pack- 
aging or budgetary constraints, and newly developed 
exercises. 
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FIELD TRIALS (TRYOUTS) - A pretest of exercises to obtain 
information regarding clarity, difficulty levels, timing, 
feasibility and special administration problems needed for 
revision and selection of exercises to be used in the as- 
sessment.. 

FOIL - See "RESPONSE OPTIONS." 

FOLLCW-UP - Special studies conducted in 1972-73 1 1975-76, 
1976-77 and 1978-79 to follow-up a subsample of in-school 
17-year-old nonrespondents to determine enrollment status 
and administer assessment packages, if possible. The 
1972-73 study was a major study of 17-year-old nonre- 
sponse , sometimes referred to as the "ffe-Show Study." 

FORCED NO RESPONSE - A machine generated no response to a 
question vAiich is inappropriate for the respondent to an- 
swer. For exanple, if a respondent answers no to a ques- 
tion, "Did jou go to high school?" then the respondent re- 
ceives a forced no response to a subsequent question "What 
was the primary emphasis of your high school courses?"\ 

GROUP ADMINISTRATION - The mode of administration where an 
exercise. booklet is administered to a group of respondents 
in a school and a paced audio tape is used to provide uni- 
form instructions and oral presentation of exercises. 
From 1969-70 to 197^-75, the target group administration 
size was 12. Since then session sizes have been allowed 
to vary, with an average target size of 16. 

HAND SCORING - The rating or categorizing of responses for 
optical scanning. Multiple-choice exercises can be di- 
rectly machine scored; however, open-ended exercises and 
individual exercises must be coded in scoring ovals so 
that they can then be machine scor,^ . See "SCORING 
GUIDE." 

I DON'T KNOW - The last response option on NAEP multiple- 
choice" exercises. Ch open-ended exercises, respondents 
are encouraged to, write "I don't know" if they do not know 
the answer. 

ID NUMBER - Any identification number. Usually refers, to 
the unique number assigned to each respondent. NAEP does 
not keep records of the names of any individuals. 
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IN-SCHOOL (IS) ASSESSMENT - The administration of assessment 
booklets to 9-.year-olds, 13-year-olds and 17-year-olds 
currently enrolled in public or private schools. 

INDIVIDUAL AEMINISTRATION - The mode of administration vAiere 
a booklet is administered to one person at a time. The 
administrator reads the exercises, using an interview for- 
mat. Although there are no set response times for the in- 
dividual exercises in the booklet, the booklets are de- 
signed to require no mOre than a total of ^5 to 50 minutes 
of a respondent's time. 

INDIVIDUAL EXERCISE - An exercise that is administered to 
only one person at a time. These are generally exercises 
that require interview techniques or active responses 
(such as singing) . 

INDIVIDUAL BOOKLET - A booklet composed of individual exer- 
cises that is administered to only one person at a time. 

INELIGIBLES - Individuals \itio do not meet the criteria for 
inclusion in the assessment. Ineligibles include those 
who fall outside the age range, those- i4io are not enrolled 
in public or private schools at the time of assessment for 
the 9-, 13- and in-school 17-year-old assessments, and 
those who are functionally handicapped so that they, cannot 
participate in the assessment, currently defined as; 



Non-English speaking persons; 

Respondents identified as nonreaders during the as- 
sessments; 

Persons physically or mentally handicapped, including 
educable mental retarded (EMR)« in such a way that 
they could not respond to NAEP exercises as they are 
normally administered; 

Students attending public and private schools estab- 
lished for the physically and/or mentally handicap- 
ped . 

ITEM - See "EXERCISE.", 



1. ' 

2. 
3. 
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INVALID EXERCISE - An exercise v*iich is technically deficit 
and omitted from analyses. IVpical problemSJ include am- 
biguous WDrding, multiple correct response choi?3ec and cu- 
ing by other exercises in the same booklet. Qpen--ended. 
exercises which elicit many vague or irrelevant responses 
are included in this category. 

ITEM BOOKLET - See "BOOKLET," 

ITEM PART - Some items have more than one part. .Generally, 
each part of an item asks a separate question. Parts may 
all pertain to one stimulus , such as a graph or a table, 
or may concern th«^ same topic. Exceptions are open-ended 
items that ask for multiple responses ("Give three rea~ 
sons...."), responses that are scored on multiple dimen- 
sions and complex Individually-administered exercises; 

ITEM VARIABLE, - See "VARIABLE." 

JACKKNIFE - The name of the algorithm used by NAEP to esti- 
mate standard errrors of percentages and other statistics. 
See i^pendix 5 for details and references. 

KEY - An indicator of v*ich value in the response range is 
considered a correct or acceptable response to the exer- 
cise. 

LEARNING AREA - One of the 10 areas assessed by the NAEP 
project: Art, Career and Occupational Development, Citi- 
zenship, Uterature, Mathematics, Music, Reading, Science, 
Social Studies and limiting. Also called "subject area." 

MARKER EXERCISE - An exercise which occurs in two or more 
booklets for an age group. The most cocnmon example is 
curriculun-related questions that appear in all booklets. 

MIGRATION REGION - From 1972-73 to 1979-80, 13- and 
17-year-old respondents were asked where they lived on 
their birthday four years earlier. (Adults were asked 
similar questions for various time points.) To meet con- 
fidentiality requirements , the data were receded to show 
whether the respondent lived in the same state, same re- 
gion, a different region (or other) compared to four years 
earlier. 
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MODAL GRADE - The grade in which the majority of each in- 
^chool age group is enrolled. The modal grades are: 

Age Modal G^ade 



MULTIPLE-CHOICE EXERCISE - An exercise (Item) with preprint- 
ed response options, as opposed to open-ended (or free-re- 
sponse or interview) exercises, where a written or verbal 
response is required. 

N-COUNT - A numeric count. Usually refers to counts of re- 
spondents. 

NAEP NUMBER - An exercise identification nunber which usual- 
ly contains coded information about the objective and/or 
the content area the exercise measures. 

OBJECTIVE - A desirable educational goal agreed on by schol- 
ars in the learning area, edxicators, and concerned lay 
persons. Cbjectives are established through a consensus 
approach^ 

OBJECTIVES REDEVELOPMENT - After the initial assessment of a 
learning area, one of the first steps in preparing for 
subsequent assessments of the learning area is a review of 
the learning area objectives from the previous assessment 
by scholars in the field, educators, and concerned lay 
persons. These reviews nay result in revision, modifica- 
tion or total rewriting of the learning area objectives to 
reflect current curricular goals and emphases; they may 
also result in the endorsement of the existing objectives. 

OPEN-ENDED EXERCISE - A non-multiple-choice exercise that 
requires some type of written or oral response. 

OUT-OF -SCHOOL (OOS OR OS) ASSESSMENT - The administration of 
assessment materials to 17-year-olds not enrolled in sec- 
ondary school. Fi-offl 1969-70 to 1 972-73 • out-of-school 
17-year-olds were located and assessed during the house- 
hold survey of young adults. The yield of 17-year-olds 
was so low that, in 1970-71. a supplementary frame assess- 
ment was implemented, using lists of dropouts and early 
graduates provided by schools participating in the in- 



9 

13 
17 



4 
8 
11 
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school assessment. Since 1973-7^, the school lists of 
dropouts and early graduates have been the only source of 
out^of-school 17-year-olds. 

OVERLAP EXERCISE - An exercise administered to more than one 
age class in the same assessment. 

P-VALUE - NAEP terminology for the percentage of responses 
to a response option or category (often the acceptable re- 
sponse option) . 

PACED AUDIO TAPE ~ A tape recording that accompanies group 
administration booklets to assure uniformity in adminis- 
tration. Instructions and exercises are read by the an- 
nouncer on the tape so that reading difficulties will not 
interfere vdth an individual » s ability to respond, ijn ex- 
ception is assessments of reading in v*iich the instruc- 
tions are read by the announcer vAiile the reading passag- 
es, questions ai)d response choices are read by the 
respondent. Response time in included on tape. See 
"GROUP AtMINISTRATIONi" 

PACKAGE - See "BOOKLET." 

PACKAGE WEIGHT - See "booklet weight" under "WEIOiT." 

PACKAGING PLAN - The process of allocating exercises select- 
ed for the assessment into various booklets and arranging 
the exercises within booklets. Ihe plan considers exer- 
cises which ,should not be packaged together, exercise or- 
der, objective/ content area coverage, exercise formats, 
difficulty, etc . 

PRIMARY SAMPLING UNIT (PSU) - First stage sampling units, 
typically a county or group of contiguous counties. 

PRINCIPALIS QUESTIONNAIRE ^ A data collection form given to 
school principals. It contains, questions about enroll- 
ments, size of community, occupational composition of com- 
munity, etc. See also "SUPPLEMENTARY PRINCIPALIS 
QUESTIONNAIRE." 

PROBE - A small scale assessment of a sp)€cific topic or 
area, usually administered to only one age group. Probes 
are generally not designed to measure change. 

PSU - See "PRIMARY SAMPLING UNIT." 
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RACE - NAEP collects racial/ethnic data by visual observa- 
tion of respondents (and since 1975-76, 17-year-olds have 
also been asked to classify themselves). Black-white-oth- 
er respondent data are available for all assessments, 
while Hispanic respondent data are available from 1971-72 
on» Since the 1976-77 adult and 1977-78 in-school assess- 
ments, standard federal categories have been used (Hispan- 
ic heritage, black, white, Asian, American Indian and 
Alaskan Native). Che inconsistency occurs in race classi- 
fication for the 1970-71 assessment. In that year Hispan- 
ic respondents are classified as whites, in other years 
Hispanic respondents are classified as other. 

RECEIPT CONTROL - Procedures implemented by scoring staff to 
check in and ^reen materials from the field. In formation 
gained from receipt control procedures is relayed to field 
administration staff so that any errors may be corrected. 

RECYCLED EXERCISES - See "SECURE EXERCISES." 

REGION - NAEP uses the geographic region definition of the 
office of Business Economics, Department of Commerce, for 
both sample stratification (1970-71 on) and reporting. 
Respondents are classified by region based on the location 
of their school or home (for household "surveys of out-of- 
school 17-year-olds and adults). The categories are: 

1 = Northeast: 



Connecticut 
Delaware 

District of 03li«ibia 

Maine 

Maryland 

Massachusetts 



Pennsylvania 
Hiode Island 
Vermont 



New Hampshire 
New Jersey 



New York 



2 = Southeast: 



Alabama 

Arkansas 

Florida 

Georgia 

Kentucky 

Louisiana 



Missis si ppi 



North Carolina 
Soiith Carolina 
Tennessee 



Virginia 
West Virginia 



3 = Central: 



Illinois 

Indiana 

Iowa 



Missouri 
Neb raska 
torth Dakota 
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Kansas Chlo 

Michigan o South Dakota 

Minnesota Wisconsin 

We St: 



Alaska Nevada 
Arizona New Mexico 

California Cklahoma 
Colorado Oregon 
Hawaii Texas 
Idaho Utah ! 

Montana Washington 

Wyoming 

RELEASE NUMBER - An . Identl flcatlon nunber assigned to an ex- 
ercise when NAEP reports are organized. The reporting 
theme to which the exercise belongs Is coded In the nun- 
ber . 

RELEASED EXERCISE - An exercise for which results and exer- 
cise text have been reported to the public. This type of 
exercise Is In the public domain and may be used by any- 
one. However, If the exercise Includes copyrighted mater- 
ial a user must obtain permission from the copyright hold- 
er to use that material. 

RELEASED EXERCISE SET - A set of the released exercises. In- 
cluding documentation and scoring guides, made available 
to state and local education agencies, the research commu- 
nity and the general public. 

REPLICATE - Replicate Is used In two distinct ways by NAEP. 
For sample selection and variance estimation purposes, the 
variance estimation replicate Is a first -stage sampling 
unit Tn the 1975-76 and subsequent assessments. Except 
for very large SMSAs (which are further stratified into 
multiple variance estimation replicates), variance estima- 
tion replicates correspond to primary sampling units. For 
purposes of allocating workload (called package assign- 
ment) , a package asslgrment replicate Is a set of schools 
In which one group administration of every exercise book- 
let Is scheduled. Variance estimation replicates contain 
one to three package assignment replicates. 

REPORTING groups' - Categories of variables for which Nation- 
al Assessment data are reported. They typically Include 
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sex, race, region, community size type of community , 
modal grade and parental education, 

RESCORE - If an open-ended exercise is affected by recent 
history, it may be necessary to make changes in the scor- 
.ing categories of the exercise when it is used to measure 
change and then to rescore the responses from the previous 
assessment using the new scoring categories. For other 
open-ended exercises, it is necessary to, have responses 
from both assessments scored simultaneously to insure com- 
parability of scoring. 

RESPONDENT - A person viio responds to the exercises and 
background questions in an assessment booklet. Ih-school 
assessment respondents respond to only one booklet; out- 
of-school 17-year-olds and adults may respond to as many 
as three booklets. ^ 

RESPONDENT WEIGHT - See "WEIfflT." 

RESPONSE OPTIONS - Different alternatives to a multiple- 
choice question that can be selected by the respondent. 
Also called "foils." 

RESPONSE TAPE - Audio tape recording of a verbal or musical 
response made by the resjx)ndent. 

REVIEW CONFERENCE - A, conference held to review the objec- 
tives of a learning' area to assure their acceptance by 
scholars educators , and lay persons or to review exercis- 
es for racial, ethnic, social or regional bias and to as- 
sure their acceptance as measures of the objectives by 
scholars, educators and lay persons, 

SAMPLE - A subset of the population who are assessed in or- 
der to estimate the performanc^e of the total population. 

SCHOOL WEIGHT - See "WEIGHT". ' 

SCORING GUIDE - A guide for hand-scoring an open-ended exer- 
cise that specifies descriptive or diagnostic categories 
by giving definitions and example responses. Categories 
are usually defined as "acceptable" or "unacceptable." 

SCORING OVALS - Scannable ovals printed beside multiple- 
choice foils and printed at the bottom of the page for 
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open-ended exercises (to be used in hand scoring). When 
the ovals are marked • they can be machine scored. 

SECURE EXERCISES - The set of exercises that is kept secure 
from one assessment to the next and used to measure chang- 
es (growth or decline) in performance for the learning 
area. Also called urireleased exercises and recycled exer- 
cises. 

SIZE AND TYPE OF COMMUNITY - The seven size and type of com- 
munity reporting categories consist of three "extreme" 
types of c<xnmunity and four "residual" community sizes. 
Each extreme category includes approximately 10% of the 
respondents at each age level; the remaining respondents 
are classified according to one of the Community Size cat- 
egories. The extreme categories are: 

.1 = Extreme Rural - Rural areas where a high proportion 
of adults are farmers or farm wrkers and a low pro- 
portion are professional, managerial or factory 
workers. At least some of the respondents are from 
open country or places less than 2,500 population, 
excluding places greater than 10,000 (ft^om the Prin- 
cipal's CJuestionnaire) and the suburbs of medium and 
large cities. These respondents must be located in 
Cbmmunity Si7-e category i| (Small Places). 

2 = Low Metro - City areas where a high proportion of 

the adult population is either n^t regularly em- 
ployed or on welfare and a low pjoportion is em- 
ployed in professional or manj^gt-rial positions. 
These respondents must be in Community Size catego- 
ries 1 or 2 (Big City or Ea.g City Fringe). 

3 = High Metro - City areas where a high proportion of 

adults are employed in professional or managerial 
positions and a low proportion are factory or farm 
workers , not regularly employed or on welfare. 
These respondents must be in Community Size catego- 
ries 1 or 2 (Big aty or Big City FY-inge) . 
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Respondents were placed into one of the extr«ne categories 
based on an occupational profile of the community in which 
the school was located. For in«-school respondents at each 
Bge and for l^he supplementary frame sample, the school 
principal provided estimates of the percentage of students 
whose parents fit into each of six different occupational 
categories. From these proportions of various occupation- 
al categories, the three extreme groups are obtained. The 
remaining respondents are classified into four "residual" 
categories according to the Community Size category in 
which the school or tousehold was located: 

H = Main Big QLty - Areas located within the limits of 
big cities with population greater than 200,000 but 
not included in either the low Metro or High Metro 
categories (Community Size category 1, less High 
Metro and Low Metro) . 

5 = Urban FY-inge - I970 Census urbanized areas of big 

cities of population greater than 200,000 but out- 
side the city limits and not classified as Ldw Metro 
or High Metro (Community Size category 2, less High 
and Low Metro) . 

6 = Mediun Cities - Cities with population between 

25,000 and 200,000, excluding metropolitan areas of 
big cities (Community Size category 3). 

7 = Small Places - Open country or places with popula- 

tions less than 25.000, excluding those classified 
as Extreme Rural ( Community Size category 4, less 
Extreme Rural) . 

Note : Initial analyses and reports for the 1969-7O to 
1972-73 assessments employed a somewhat different defini- 
tion of Size and lype of Cbmmunity. Since late 1975f all 
analyses and reports involving those asses^ents have uti- 
lized the current definition. 

SMSA - Standard metropolitan statistical area, an economic 
and social unit defined by the U. S. Census Bureau. 

SOURCE DOCUMENTS - H«3terials containing original respondent 
responses such as booklets or workbooks or containing 
original background information from other sources such as 
schools. 



109 



PACE 3- 17 



APPENDIX 3 

STANDARD ERROR - A measure of sampling variability for a 
statistic. Because of NAEP'S complex sample design » 
standard errors are estimated by jackknifing first stage 
sample estimates. 

STANDBY SCHOOLS - Schools with too few age-eligibles to ac- 
comodate at least one group administration of a booklet 
are called standby schools. They are typically stratified 
and sampled differently than schools with more age-eligi- 
bles. 

STEM - The portion of an exercise or exercise part that 
states the problem or asks the question. 

STIMULUS - The task which is presented to the respondent. 
This includes one or more written or oral questions (sae 
STEM). It may also include additional material such as a 
reading passage, map, charts or graph, photograph or pic- 
ture, musical selection, oral reading, a physical object 
and so forth. 

STOC See "SIZE AND TYPE OT* COMMUNITY." 

SUBJECT AREA - See "LEARNING AREA." 

SUBPOPULATION OR SUBGROUP - Groups, such as males and fe- 
males, within the national population for vhich results 
are reported . 

SUPPLEMENTARY FRAME - Used to locate out-of-school 
•?7-year-olds , the supplementary frame consists of lists cf 

dropouts and early graduates obtained from a subsample of 

schools included in that year's assessment sample. (See 
also "OUT-OF-SCHOOL ASSESSMENT.") 

SUPPLEMENTARY PRINCIPAL'S QUESTIONNAIRE - A data collection 
form given to" principals. They are asked to respond to 
questions concerning course offerings, materials, and 
staffing specific to the learning area(s) being assessed . 
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TARGET POPULATION - Individuals who meet the criteria for 
inclusion in the assessment, including those falling with- 
in the age range, enrolled in public or private schools at 
time of the 9-, 13- and in-school 17-year-old assess- 
ments, and functionally able trO participate in the assess- 
ment. 

\ TAILS HEET - The last page of questions in an in-school as- 
/ sessment booklet. Respondents provide personal background 
information about their home environment, parents' educa- 
tion levels, etc. 

TAPESCRIPT - A script prepared for the announcer to use in 
producing the paced audio tape. It indicates exactly what 
is to be read and the amount of response time to be al- 
lowed for each exercise. See "PACED AUDIO TAPE." 

TIMING - Most NAEP exercises are administered with a paced 
audio tape to standardize data collection conditions, In- 
cluding the amount of time allowed to respond to each ex- 
ercise. 

TOC OR TYPE OF COMMUNITY - The three extreme Size and IVpe 
of Community categories. 

UNIVERSAL - A^lassification category vrfiich identifies exer- 
cise parts included in an overall sunmary of a content 
area, for exanple a sunmary of citizen ship/ social studies 
performance. In some cases it identifies exercise parts 
that > are included in any type of sunmary for an area, for 
example sunmaries of cognitive science performance and af- 
fective science performance. 

UNRELEASED EXERCISE - See "SECURE EXERCISE." 
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VARIABLE - Any data field on National Assessment data files. 
Different convention? and docunentation standards -apply to 
exercise (item) variables than to background (including 
demographic and housekeeping) variables. NAEP docunents 
often use the- term variable as if it applied only to re- 
porting variables, such ss region, sex, and race. 

WEIGHT OR SAMPLING WEIGHT - National Assessment uses deeply 
stratified, multistage probability sample designs with 
differential sampling rates for various subpopulations. 
The selection probability of each respondent is calculated 
and its reciprocal (adjusted for nonresponse) is used to 
weight each response in statistical calculations. The 
weights compensate for unequal sampling rates and insure 
proper representation in the population structure. Na- 
tional Assessment data files include several distinct 
types of weights, which differ primarily in the way non- 
response adjustments are made. The booklet (or respon- 
dent or student ) weight is the reciprocal of the probabil-^ 
ity of the respondent being selected for a ' particular 
exercise booklet, adjusted for nonresponse to that book- 
let. This weight is used by National Assessment in virtu- 
ally all statistical analyses. The multi-booklet analysis 
weight is the reciprocal of the probability of selecting a 
respondent for any booklet, adjusted for nonresponse to 
all booklets. The multi-booklet analysis weight can be 
used for analyses across all booklets, but the booklet 
weight works equally well for either booklet-specific or 
cross-booklet analyses. Beginning with the 1979-80 as- 
sessment, an additional school-based weight has also been 
included. Ihis weight is the inverse of the probability 
of selecting a school, adjusted for school nonresponse . 
It is used for school-level analyses v*iere the school is 
considered the unit of analysis. 

WORKBOOK - In some assessments (1972-73 and all art assess- 
ments), respondents were asked to answer certain items in 
a separate booklet, called a vorkbook. 
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INFERENCE FROM SURVEY DATA AND ANALYSIS OF DATA 
FROM COMPLEX SAMPLE SURVEYS 



NAEP is a sample survey which collects data by . using a 
multistage design with unequal probabilities of selection of 
elements. Such a survey design is commonly referred to as 
"complex" sampling. It permits the collection of represen- 
tative data for the population and many subgroups in an ex- 
tremely cost effective manner • NAEP data are suited for 
various descriptive purposes. 

1. Estimation of achievement by exercise and sunmary 
level for the age populations as a whole and for a 
.variety of subgroups of the populations. 

2. Estimation of changes in achievement, on a variety of 
tasks (and sunmaries) over time. Chfenge can be. meas- 
ured for populations and for subgroups. 

3. Explorations jf observed associations between vari- 
ables of interest. 

The various descriptive statistics should be computed 
taking the structure of the sample into account. 

The NAEP data present certain difficulties in analysis, 
which are shared by all complex surveys. These difficulties 
include problems in inference due to the type X}f data and 
complications in analysis due to the sample design. These 
difficulties are detailed in the following sections. 
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Inference From Survey Data 

Because the data collected Troai a survey are observation- 
al and .do not result from the control of variables (as do 
data collected from a designed experiment) they require 
careful interpretation. In experimental research it is of- 
ten possible to infer causation because the researcher has 
controlled the individual variables. In surveys, because 
the values of the various variables are not controlled and 
because the variables tend to be related in value to each 
other and to other extraneous variables, such control is 
generally not possible. Because correlation, in itself, 
does not prove causation, the direction of causation is of- 
ten unclear for many variables from survey data, such as be- 
haviors and correlated attitudes. 

In applying standard statistical tests, it is often the 
case that undesirable confounding occurs between pertinent 
variables and other perhaps extraneous, variables; hence, 
tests performed in the presence of these, often unremovable, 
confounding variables require very careful interpretation. 
Because of this, NAEP data should be examined in an explora- 
tory, rather than confirmatory, framework. 



Complications in Analysis Due to Sample Design ^ 

Complications in analysis occur because of the multistage 
design of the NAEP sample. Many standard statistical proce- 
dures assune that data are acquired by means of a simple 
random sample of the population and that individuals are in-' 
dependent. Because the NAEP sample employs stratification, 
clustering and unequal probabilities of selection , these as- 
sumptions are not met. 

Certain subgroups of the population are sampled at a 
higher rate than the remainder of the population in order to 
ensure adequate representation. Consequently, those sub- 
groups, which tend to have different characteristics (in- 
cluding achievement) than the remainder of the population, 
are over-represented in the sample. Analyses which ignore 
this are apt to produce biased and misleading information, 
since those groups may have unwarranted impact . This diffi- 
culty is avoided by conducting weighted analysed, in vhich 
the weight assigned to an individual is related to the re- 
ciprocal of his or .her probability of selection. 
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Because of cost and administrative efficiency considera- 
tions, 'NAEPdata are obtained by selecting a nunber of 
schools and then selecting a nunber of students within each 
of the schools. Since the students are selected in clus- 
ters, observations from various students are not indepen- 
dent. Student responses within a school tend to be rela- 
tively more homogeneous than student reaponses from 
different schools. 

Ignoring the effect of stratificiation and clustering in 
analysis tends to produce severe underestimates of the var- 
iablity of statistics. Many studies (such as Ross (1976), 
jash and F^ankel (197^), R*ankel (1971)) have demonstrated 
large influences of complex survey designs on sampling er- 
rors of various statistics, such as regression and correla- 
tion coefficients. It has been demonstrated (Shah, Itolt and 
Folsom (1977)) that regression analyses of data from complex 
survey samples produce tests of significance i*iich are gen- 
erally too liberal vrfien the structure of the sample ia not 
taken into account. Additionally, as noted by Fellegi 
(1979) in his consideration of goodness of fit tests, the 
distribution of certain statistics, as well as their disper- 
sions, can be affected by the sample design. 

Because of the nonlinearity of many of the statistics of 
interest, it is' not currently possible to exactly dccount-:-. 
for the sample structure in analysis. ^Fbwever, several pro- 
cedures exist which approximately do this. An ong' these are: 

1. jackknifing — the procedure used by NAEP (detailed by 
Folsom (1977)) ; . f 

2. balanced repeated replications (detailed by McCarthy 
(1969)); and 

3. Taylor series approximation (detai^led by Folsom 
(1977))- \ 

Shah, tolt and Folsom (1977) give procedures for estimation 
and hypothesis testing of regression models for data from a 
complex sample survey using Taylor series approximations. A 
general procedure for obtaining approximate variances by 
Taylor series approximations is given by Wbodruff and Cansey 
(1976). 
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Fellegi (1979) and McCarthy ( 19695 ^give procedures for 
using balanced repeated replications to conduct goo(iness of 
fit tests. 

Folsom (1977) gives the procedures used by NAEP in its 
analyses using jackknifing methodology. 

A comparison of the performance of the three procedures 
for means, variances, correlations, and regression coeffi- 
cients is given by Frankel (1971) and Kish and FY-ankel 
(1974). 



Approximately Accounting for Sample Design vdth Design 

Effects 

It may be possible to approximately account for the ef- 
fects of the sample design by using an inflation factor, the 
design effect, developed by Kish ( 1967) and extended by Kish 
and R-ankel (1974). Ihe design effect for a statistic is 
the ratio of the actual variance of the statistic (taking 
the sample. design into account) over the variance assuning a 
simple random sample with the same nqnber of elements. The 
design effect may be used to adjust error estimates based on 
simple random sampling assunptions so that the effect of the 
design is approximately accounted for. In practice, this is 
often accomplished by dividing the total sample size by the 
design effect and then using this effective sample size in 
the computation of errors. It must be kept in mind, howev- 
er, that the value of the design effect :deperids on the vari- 
ables considered in a particular analysis as well as the 
clustering effects occurring among sampled elements. 

,1. Based on empirical results and theoretical considera- 
tions, Kish and Frankel (1974) have developed several 
conjectures about design effects: 

2. Generally, the design effects for complex statistics 
from comple:-r samples are greater than one and so var- 
iances based on simple random sampling assunptions 
tend to be underestimates. 
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3. The design effects for complex statistics (such as 
regression coefficients) tend to be smaller than the 
corresponding design effects for means of the same 
vari-ables. Hence, the latter estimates, which are 
more easily computed, tend to give over-estimates of 
.the design effects of complex statistics. 

i4. The design effects of complex statistics tend to re- 
semble those of means, variables with a high design 
effect for the mean tend to also have high design ef- 
fects for complex statistics involving those vari- 
ables. 

. National Assessment has computed design effects for vari- 
ous types of statistics. A key statistic that has been ex-- 
amined is tKe estimated performance, P, of a subgroup of the 
population on an assessment exercise,. This estimate, which 
is a weighted mean of the responses of individuals in the 
subgroup to the exercise, has a design effect of the form 

deff(P) = Var (P)/(P(1-P)/N) . 

In the above, N is the total nunber of sampled individu- 
als in the. subgroup who responded to the exercise and Var(P) 
is the jackknifed variance of P (vrfiich takes the sample de- 
sign into account). Ijpon examination of the- distributions 
of design effects across exercises within reporting category 
(i.e., whites, low metropolitan , northeast) for a variety of 
ages, subjects and years, two things were noted: 

1. The various distributions were remarkably similar, 
although there were relatively more large design ef- 
fects for the smaller, more clustered, subgroups such 
as low metropolitan . 

2. The centers of all distributions were close to two', 
with the majority of values being less than two. 

Because of this, a design effect of two was deemed a rea- 
sonable value to use for complex analyses. Based on Kish 
and Fi*ankel»s conjectures, this value should be an approxi- 
mate upper bound for design effects of complex statistics. 
In a small scale regression analysis on NAEPdata, the mean 
value of the design effects of regression coefficients was, 
in fact, two. * 
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National Assessment has conducted regression analyses 
using design effects to account for the sample design and 
has found them adequate for exploratory purposes. The pro- 
cedure used by National Assessment JTor regression analysis 
(called standard regression) using staiidard stati stical 
packages (such as SPSS, BMDP and SAS) is as follows: 

!• Every individual has a sampling weight which gives 
the relative importance of that individual in the 
population. These weights are necessary to provide 
proper contributions of the various subgroups in the 
estimates. 

2. Each individual is assigned a scaled weight which is 
the person's sampling weight times the factor f r 
(N/2)WT0T. In the formation of f , N is the sample 
size, N/2 is the effective sample size, and WTOT is 
the total of the sampling weights of all N individu- 
als in the analysis. The scaled weights sun to the 
effective sample\size and maintain the relative 
weightings of the ir^ividuals. 

3. A weighted regression\nalysis is then conducted us- 
ing, scaled weights. 

The estimates of the regression parameters using this 
procedure are identical to those obtained, using the mor'e 
complicated Taylor series and jackknifing procedures given 
by Folsom (1977). Although the estimated variances of the , 
parameter estimates obtained using the above procedure dif- 
fer from the more exact variance estimates (from Taylor se- 
ries and jackknifing), they tend to be similar and appear 
adequate for exploratory purposes. Ifowever, significance 
tests using the standard regression approach need to be mod- 
ified. The actual nunber of degrees of freedom for error is 
approximately the nunber of variance estimation replicates 
minus the nunber of strata rather than the effective sample 
size minus the nixnber of parameters, the value given by the 
standard analysis. Therefore, significance should be as- 
sessed by comparison to criticai values with the smaller er- 
ror degrees of freedom. 
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It should also be noted that the estimated covarlancest 
between regression parameters using the standard regression 
can differ substantially from those estimated using Taylor 
series and jackknifing. Ihis is because the joint distribu- 
tions of subgroup performance and subgroup s^ize are account- 
ed for in the Taylor series and jackknifing Wocedui'es while 
they are not in the standard regression. 1 

Tests of goo^Jness of fit and independence\can be per- 
formed on NAEP data by using the design effect approach. To 
accomplish this, the counts of individuals vd/thin various 
cells are replaced by sums of scaled weights wnere, again, 
the sum across all subjects of the scaled weights, is the ef- 
fective sample size, i^art from the substilcution of scaled 
weight totals* the analyses proceed in the usual manner* 
Felegi (1979) has empirically investigated this approach and 
found that although the tests are somewhat liberal this^ap- 
proach. peforms acceptably well for survey designs such as 
National Assessment's. 
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NATIONAL ASSESSMENT ESTIMATION OF STANDARD 
ERRORS 



Several measures of achievement that National Assessment 
uses in its reports are described in the Data Analysis sec- 
tion of this docunent. Ihe sample design is a complex, 
deeply stratified, multistage probability sample design. A 
reasonably good approximation of standard error estimates of 
these achievement measures can be obtained by applying the 
Jackknife procedure to first-stage sampling units (repli- 
cates) vdthin strata, using the method of successive differ- 
ences and accunulating across strata. 

s 

In this appendix, the measures of achievement are first 
defined in algebraic form, followed by a description of the 
Jackknife method used by National Assessment to estimate 
their standard errors. 



Measures of Achievement 

Based on the sample design, a weight is assigned to every 
^.ndividual who responds to an exercise administered in an 
assessment.' The weight is the reciprocal of the probability 
of selecting a particular individual to take a particular 
exercise with adjustment for non-response. Since the proba- 
bilities of selection are based on an estimated nunber of 
people in the 'target age population, the weight for an indi- 
vidual estimates* the nunber of similar people that that in- 
dividual represents in the age population. As explained in 
Appendix 2 the weights were adjusted to reflect information 
from previous assessments on population distributions* 

. A sun of the weights for all individuals at an age level 
responding to an exercise is an estimate of the total nunber 
o^f people in that age population. A sun of weights for all 
individuals at an age responding correctly to an exercise is 
an estimate of the nunber of people who would be able to re- 
spond correctly in the age population if the entire popula- 
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tlon were aaaeased. These concepts alio apply to any 
reporting group (e.g., defined by region, sex t etc.) and 
category of response (e.g., correct, incorrect and "I don't 
know") . 

e 

Let W = sum of weights for respondents to exercise e who 
ihk 

are in reporting subgroup i and who arie in the 

kth renlicate of the hth sampling stratun, and 

ej 

C = sum of ights for respondents to exercise e who 
ihk 

are i orting subgroup i , who are in the kth 

repliC' '^f the hth sampling stratun, and \i\o 
selected , esponse category j (e.g . , correct 
foil) for the exercise. 

e ej 
Note that W = Sum C 

ihk j ihk 

Then, sunming k over the n sample replicates in the 

h 

stratun h, and sunming over the H sampling strata, 
n 

e H h e^ 

W = Sum Sum W estimates the nunber of eligibles in 

i++ h=1 k=1 ihk 
the population who are in subgroup i . 

n - 

ej H h ej 
Similarly, C s Sum Sum C 

i++ h=1 k=1 ihk 

estimates the nunber of eligibles in the population v*io are 
in subgroup i and who would select response category j for 
exercise e. s 

An estimate of the proportion*'Of the eligibles in the age 
population in group i who would select response category j 
on exercise e is: 
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ej ei e 

C / W , 
i i++. 

In the special case where the proportion of all age eli- 
gibles who would select response category j on exercise e is 
estimated » the index A (for ALL) will used in place of i 
as follows: %v 

>. 

ej ej e 

(2) P = C / 

A A-M-' A«f+, 

In ' National Assessment reports, the proportion in (1) 
multiplied by 100 is called the group percentage, and the 
proportion in (2) multiplied by 100 is called the national 
percentage. The difference between the proportion in sub- 
group i who would select category j on exercise e and the 
proportion in the nation is denoted by: 

ej ej ej 

(3) dP s P - P . 

i i A 

National Assessment also reports the arithmetic mean of 
the percentage of correct responses over sets of exercises 
corresponding to the measures in (1), (2) and (3). These 
means are taken over the set of all exercises or a subset of 
exercises classified by a reporting topic or content objec- 
tive. The mean percentage of correct responses taken over m 
exercises In some set of exercises corresponding to measures 
(1), (2) and (3) are, respectively: 

^ e e 

(4) P = - Sum C / W , 

i m e i++ 1++ 



_ 1 e e 

(5) P = - Sum C / W and 
A m e A+-f 1++ 



It 



(6) dP = P" - P 
1 i 
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Note that the response category subscript j has been 
suppressed since the means are uncierstood to be taken over 
the correct response category for each exercise. 

Each of these six achiever^ent measures is computed and 
routinely used in reports describing achievement data for 
any assessment. Ihe simple difference in these measures be- 
tween two assessments of the same exercise ( or sets of iex- 
ercises) provides six measures of change in achievement that 
are routinely used in National Assessment's change reports. 
The next section describes how standard errors are estimated 
for the 12 statistics routinelyp^ed in jfeEP reports. 



Computation of Standard Errors 

In order to obtain an approximate measure of the sampling 
variability in the statistics (1) through (6), a jackknife 
replication procedure for estimating the sampling variance 
of non-linear statistics from complex, multistage samples 
was tailored to National Assessment's sample design. Miller 
(1968f 1974) and MDsteller and lUkey (1977) provide informa- 
tion about. the Jackknife technique, while reference (1) de- 
scribes how the procedure is used in estimating standard er- 
rors for National Assessment's sample designs; 

To demonstrate the computational aspects of this techni- 
que, consider estimating the variance of the statistic in 
(1) - the proportion of age-eligibles in subgroup i who 
would select response category j on exercise e. 

This statistic is based on the data from all the n 

h 

ej 

replicates in the H strata. Let p be defined as a 
<? i-hk 
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ej 

replication estimate of p and constructed from all the 

i 

replicates excluding the data from replicate k in stratutn h. 
These replication estimates are computed as if the excluded 
replicate had not responded and a reasonable nonresponse 
adjustment is used to replace the data in replicate hk in 

estimating p . Several choices for repJ^Ksing the data in 
i 

replicate hk are available. In order to obtain a convenient 
and computationally efficient algorithm for approximating 

ej e 

standard errors. National Assessment replaces C ^and W _ 

ihk ihk 

from the hk(th) replicate with corresponding suns from 
another paired replicate in the same stratum. The replicate 
estimate is then computed . The replicate estimates to 
be used in the calculations are determined by arranging 

all the replicates in each stratun into successive pairs. 
That is, replicate 1 is paired vdth replicate 2, replicate 
2 with replicate 3. 3 with ... (n - with n and 
replicate n '*ith replicate 1. • h h 

h 

ej 

The contribution to the variance of p by each pair of 

i 

replicates is the change in the value of the statistic 
incurVed by replacing the data from each replicate in the 
pfl | -tp Vri t.h the data from the other replicate in the pair and 
V\ ej 

recomputing p in the usual way. This produces two 

^ i . 
replicate estimates. Squaring the difference between these 
replicate estimates and then dividing ■ by 8 measures the 
contribution of this pair .of replicates to. the total 
variance. The sun of these contributions over all n 

h 

successive pairs in the stratun is the contribution by 
stratun h to the total variance. The square root bf the aun 
of the H stratun contributions * is the estimate of the 

ej 

standard error of p . 

i ^ 
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Algebraically, the two replicate estimates for the pair k. 



ej 

(7) P 



1 • • An 


and n 4-1 


si) are: 




h ' 




eJ 


eJ 


ej 


C 


- C + 


C 


i++ 


ihk 


ih(k+1) 


eJ 


eJ 


ej 


W 


- W + 


W 


i++ 


ihk 


ih(k+1) 



and 



eJ 

(8) P 



eJ 


eJ 


ej 


C 


- C 


+ C 


i++ 


ih(k+1) 


ihk 


eJ 


eJ 


ej 


W 


- W 


+ W 


i++ 


ih(k+1) 


ihk 



The contribution to the total variance from stratun h is: 

n ( )2 
ej 1 h ( eJ ej ) 

(9) var (P ) = - Sum (P - P ) 

ih 8 k ( i-hk i-h(k+1)) 

ej 

And, finally, an estimate of the standard error of p is: 

i 

ej H ej 1/2 

(10) SE (P ) = (Sum var P ) 

i h • ih' 

ej 

Multiplying p by 100 yields the percentage of response 
i 

ej 

to category j. Multiplying SE(p ) by 100 yields the 

i 

corresponding estimated standard error of the percentage* 
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1 

In general, the jackknifed standard errors . of the 
proportion estimates will be larger than the simple random 

1/2 ej 
sampling formula (pq/n) , where p=p .qsl-p and n is the 

i 

nunber of sampled respondents in subgroup i who took the 

ej 

exercise. The larger si ze of SE (p ) reflects mainly the 

i 

loss of precision due to cluster.^sampling of schools and 
students (see Appendix ^ for further discussion of this 
point) . 

The standard errors for the achievement measures (2) 
through (6) are computed through a series of steps analogous 

to those followed in computing SE (p ) . The most 

i 

complicated step in computing standard errors occurs in 
forming the paired replicate estimates analogous to (7) and 
C8) for each successive pair of replicates. Once this 
bookkeeping chore is done, the computations (9) and (10) 
follow in a straightforward manner. 

The standard errors for the differences bejbween tWD as- 
sessments for any of the achievement measures (1) through 
(6) are computed as the square root of the sun of the 
squared standard errors from edch of the separate assess- 
ments. 

The size of the standard errors depends largely not only 
on the nunber of replicates and schools included in the sam- 
ple, but also on the nunber of respondents in each of the 
reporting groups. Eichibit 5*1 jshows the average proportion 
of students responding to an exercise booklet for each of 
the reporting groups 'for each age. The proportions in Ex- 
hibit 5.1 are for the 1979-80 assessment, averaged over 
booklets and smoothed over several adjacent assessments. 
Figures for any particular booklet can be expected to devi- 
ate somewhat from Exhibit 5-1- 

The size of the standard errors of the means of the 
achievement measures for sets of exercises is also influ- 
enced by the nunber of exercises in the exercise set and the 
nunber of booklets _pver which the items in the set are 
spread . 
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EXHIBIT 5.1 



Reporting Groups for Ih-Sehool Students Ages .9, 13 and 17 



Reporting Groups 1979-1980 

Age 9 Age 13 Age 17 



Male 


.50 


• 50 




Female 


.50 


. 50 


.52 


Race 








White 


• 79 


• oO • 


.83 


Black 




.13 


• 12 


Other 


.07 


• 07 


.05 


Region 






• CO 


Northeast 


• 25 


Southeast 


• 22 


• 23 


. 20 


Central 


• 27 


.27 


.29 


West 


• 26 


• 25 


- .26 


Parental education 








Not graduated high school 


.09 


. 13 


• 15 


Graduated high school 


.2'» 


.32 


•32 


Post high school 


• 33 


• 12 


• 18 


Uiknown 


• 34 


• 13 


• 05 


lype of community 






.08 


Extreme rural 


. •OS 


. 10 


Low metro 


.07 


• 07 


• 09 


' High metro 


.11 


• 11 


.11 


Other 


.71* 


• 72 


• 72 


Size of community 








Big city 


.20 


• 21 


• 19 


Fringes around big cities 


• 22 


.22 


• 26 


Medium city 


.12 


• 11 


.11 


analler places 


.H6 


•ne 


• 114 


a*ade in school 








<3. <?• 10 


<.01 


• 02 


.02 


3» ?• 10 


.23 


• 25 


• 13 


8, 11 


• 75 


^72 


• 75 


>8t 12 


<^01 


<^01 


• 10 


: Other 


<.01 


<^01 


<.01 



